Xpath error invalid expression

XPath and XSLT with lxml

lxml supports XPath 1.0, XSLT 1.0 and the EXSLT extensions through libxml2 and libxslt in a standards compliant way.

The usual setup procedure:

XPath

lxml.etree supports the simple path syntax of the find, findall and findtext methods on ElementTree and Element, as known from the original ElementTree library (ElementPath). As an lxml specific extension, these classes also provide an xpath() method that supports expressions in the complete XPath syntax, as well as custom extension functions.

There are also specialized XPath evaluator classes that are more efficient for frequent evaluation: XPath and XPathEvaluator. See the performance comparison to learn when to use which. Their semantics when used on Elements and ElementTrees are the same as for the xpath() method described here.

The .find*() methods are usually faster than the full-blown XPath support. They also support incremental tree processing through the .iterfind() method, whereas XPath always collects all results before returning them. They are therefore recommended over XPath for both speed and memory reasons, whenever there is no need for highly selective XPath queries.

The xpath() method

For ElementTree, the xpath method performs a global XPath query against the document (if absolute) or against the root node (if relative):

When xpath() is used on an Element, the XPath expression is evaluated against the element (if relative) or against the root tree (if absolute):

The xpath() method has support for XPath variables:

Namespaces and prefixes

If your XPath expression uses namespace prefixes, you must define them in a prefix mapping. To this end, pass a dictionary to the namespaces keyword argument that maps the namespace prefixes used in the XPath expression to namespace URIs:

The prefixes you choose here are not linked to the prefixes used inside the XML document. The document may define whatever prefixes it likes, including the empty prefix, without breaking the above code.

Note that XPath does not have a notion of a default namespace. The empty prefix is therefore undefined for XPath and cannot be used in namespace prefix mappings.

There is also an optional extensions argument which is used to define custom extension functions in Python that are local to this evaluation. The namespace prefixes that they use in the XPath expression must also be defined in the namespace prefix mapping.

XPath return values

The return value types of XPath evaluations vary, depending on the XPath expression used:

• True or False, when the XPath expression has a boolean result
• a float, when the XPath expression has a numeric result (integer or float)
• a ‘smart’ string (as described below), when the XPath expression has a string result.
• a list of items, when the XPath expression has a list as result. The items may include Elements (also comments and processing instructions), strings and tuples. Text nodes and attributes in the result are returned as ‘smart’ string values. Namespace declarations are returned as tuples of strings: (prefix, URI).

XPath string results are ‘smart’ in that they provide a getparent() method that knows their origin:

• for attribute values, result.getparent() returns the Element that carries them. An example is //foo/@attribute , where the parent would be a foo Element.
• for the text() function (as in //text()), it returns the Element that contains the text or tail that was returned.

You can distinguish between different text origins with the boolean properties is_text, is_tail and is_attribute.

Note that getparent() may not always return an Element. For example, the XPath functions string() and concat() will construct strings that do not have an origin. For them, getparent() will return None.

There are certain cases where the smart string behaviour is undesirable. For example, it means that the tree will be kept alive by the string, which may have a considerable memory impact in the case that the string value is the only thing in the tree that is actually of interest. For these cases, you can deactivate the parental relationship using the keyword argument smart_strings.

Generating XPath expressions

ElementTree objects have a method getpath(element), which returns a structural, absolute XPath expression to find that element:

The XPath class

The XPath class compiles an XPath expression into a callable function:

The compilation takes as much time as in the xpath() method, but it is done only once per class instantiation. This makes it especially efficient for repeated evaluation of the same XPath expression.

Just like the xpath() method, the XPath class supports XPath variables:

This supports very efficient evaluation of modified versions of an XPath expression, as compilation is still only required once.

Prefix-to-namespace mappings can be passed as second parameter:

Regular expressions in XPath

By default, XPath supports regular expressions in the EXSLT namespace:

You can disable this with the boolean keyword argument regexp which defaults to True.

The XPathEvaluator classes

lxml.etree provides two other efficient XPath evaluators that work on ElementTrees or Elements respectively: XPathDocumentEvaluator and XPathElementEvaluator. They are automatically selected if you use the XPathEvaluator helper for instantiation:

This class provides efficient support for evaluating different XPath expressions on the same Element or ElementTree.

ETXPath

ElementTree supports a language named ElementPath in its find*() methods. One of the main differences between XPath and ElementPath is that the XPath language requires an indirection through prefixes for namespace support, whereas ElementTree uses the Clark notation ( name) to avoid prefixes completely. The other major difference regards the capabilities of both path languages. Where XPath supports various sophisticated ways of restricting the result set through functions and boolean expressions, ElementPath only supports pure path traversal without nesting or further conditions. So, while the ElementPath syntax is self-contained and therefore easier to write and handle, XPath is much more powerful and expressive.

lxml.etree bridges this gap through the class ETXPath, which accepts XPath expressions with namespaces in Clark notation. It is identical to the XPath class, except for the namespace notation. Normally, you would write:

ETXPath allows you to change this to:

Error handling

lxml.etree raises exceptions when errors occur while parsing or evaluating an XPath expression:

lxml will also try to give you a hint what went wrong, so if you pass a more complex expression, you may get a somewhat more specific error:

During evaluation, lxml will emit an XPathEvalError on errors:

This works for the XPath class, however, the other evaluators (including the xpath() method) are one-shot operations that do parsing and evaluation in one step. They therefore raise evaluation exceptions in all cases:

Note that lxml versions before 1.3 always raised an XPathSyntaxError for all errors, including evaluation errors. The best way to support older versions is to except on the superclass XPathError.

lxml.etree introduces a new class, lxml.etree.XSLT. The class can be given an ElementTree or Element object to construct an XSLT transformer:

You can then run the transformation on an ElementTree document by simply calling it, and this results in another ElementTree object:

By default, XSLT supports all extension functions from libxslt and libexslt as well as Python regular expressions through the EXSLT regexp functions. Also see the documentation on custom extension functions, XSLT extension elements and document resolvers. There is a separate section on controlling access to external documents and resources.

Due to a bug in libxslt the usage of elements=»*»/> in an XSLT stylesheet can lead to crashes or memory failures. It is therefore advised not to use xsl:strip-space in stylesheets used with lxml.

XSLT result objects

The result of an XSL transformation can be accessed like a normal ElementTree document:

but, as opposed to normal ElementTree objects, can also be turned into an (XML or text) string by applying the bytes() function ( str() in Python 2):

The result is always a plain string, encoded as requested by the xsl:output element in the stylesheet. If you want a Python Unicode/Text string instead, you should set this encoding to UTF-8 (unless the ASCII default is sufficient). This allows you to call the builtin str() function on the result ( unicode() in Python 2):

You can use other encodings at the cost of multiple recoding. Encodings that are not supported by Python will result in an error:

While it is possible to use the .write() method (known from ElementTree objects) to serialise the XSLT result into a file, it is better to use the .write_output() method. The latter knows about the tag and writes the expected data into the output file.

Stylesheet parameters

It is possible to pass parameters, in the form of XPath expressions, to the XSLT template:

The parameters are passed as keyword parameters to the transform call. First, let’s try passing in a simple integer expression:

You can use any valid XPath expression as parameter value:

It’s also possible to pass an XPath object as a parameter:

Passing a string expression looks like this:

To pass a string that (potentially) contains quotes, you can use the .strparam() class method. Note that it does not escape the string. Instead, it returns an opaque object that keeps the string value.

If you need to pass parameters that are not legal Python identifiers, pass them inside of a dictionary:

Errors and messages

Like most of the processing oriented objects in lxml.etree, XSLT provides an error log that lists messages and error output from the last run. See the parser documentation for a description of the error log.

Note that there is no way in XSLT to distinguish between user messages, warnings and error messages that occurred during the run. libxslt simply does not provide this information. You can partly work around this limitation by making your own messages uniquely identifiable, e.g. with a common text prefix.

The xslt() tree method

There’s also a convenience method on ElementTree objects for doing XSL transformations. This is less efficient if you want to apply the same XSL transformation to multiple documents, but is shorter to write for one-shot operations, as you do not have to instantiate a stylesheet yourself:

This is a shortcut for the following code:

Dealing with stylesheet complexity

Some applications require a larger set of rather diverse stylesheets. lxml.etree allows you to deal with this in a number of ways. Here are some ideas to try.

The most simple way to reduce the diversity is by using XSLT parameters that you pass at call time to configure the stylesheets. The partial() function in the functools module may come in handy here. It allows you to bind a set of keyword arguments (i.e. stylesheet parameters) to a reference of a callable stylesheet. The same works for instances of the XPath() evaluator, obviously.

You may also consider creating stylesheets programmatically. Just create an XSL tree, e.g. from a parsed template, and then add or replace parts as you see fit. Passing an XSL tree into the XSLT() constructor multiple times will create independent stylesheets, so later modifications of the tree will not be reflected in the already created stylesheets. This makes stylesheet generation very straight forward.

A third thing to remember is the support for custom extension functions and XSLT extension elements. Some things are much easier to express in XSLT than in Python, while for others it is the complete opposite. Finding the right mixture of Python code and XSL code can help a great deal in keeping applications well designed and maintainable.

Profiling

If you want to know how your stylesheet performed, pass the profile_run keyword to the transform:

The value of the xslt_profile property is an ElementTree with profiling data about each template, similar to the following:

Note that this is a read-only document. You must not move any of its elements to other documents. Please deep-copy the document if you need to modify it. If you want to free it from memory, just do:

Источник

Python XPath SyntaxError: invalid predicate

Я пытаюсь парсить xml наподобие

Но я получаю ошибку

Что не так с моим xpath?

Follow up

Спасибо falsetru, ваше решение сработало. У меня есть follow up. Теперь, я хочу получить все элементы параграфа, которые идут перед параграфом с текстом GHF . Так вот в данном случае мне нужен только элемент XBV . Я хочу игнорировать элементы ash и lplp . Я догадываюсь один способ это сделать был бы

Но есть ли лучший способ это сделать?

2 ответа

Когда я запускаю ‘python setup.py sdist’, то всегда встречаюсь с ошибкой ‘SyntaxError: invalid syntax’. Ниже мой исходный код: printlist.py def printlist(the_list): for eachitem in the_list: print(eachitem) setup.py from distutils.core import setup setup(name=’printlist’,version=’1.0′,py_modules =.

Хочу парсить HTML с lxml с помощью XPath выражения. Моя проблема заключается в совпадении для содержимого тега: Например, дан элемент Example Я могу сопоставить атрибуту href с помощью .//a[@href=’http://something’] А вот дано выражение .//a[.=’Example’].

Как упомянул @falsetru, ElementTree не поддерживает text() предикат, но он поддерживает совпадающий дочерний элемент по тексту, поэтому в данном примере есть возможность искать элемент page у которого есть paragraph с конкретным текстом, с помощью пути ./pages/page[paragraph=’GHF’] . Проблема здесь в том, что в a page есть несколько тегов paragraph , поэтому пришлось бы итерировать для конкретного paragraph . В моем случае мне нужно было найти version a dependency в maven pom.xml, а там только единственный version child поэтому сработало следующее:

Похожие вопросы:

Я пытаюсь найти элементы в xml с помощью xpath. Это мой код: utf8_parser = etree.XMLParser(encoding=’utf-8′) root = etree.fromstring(someString.encode(‘utf-8’), parser=utf8_parser) somelist =.

import xml.etree.ElementTree as ET tree = ET.parse(‘test.xml’) xpathobjects = tree.findall(.//BuildingNodeBase[name = ‘Building name’]) Я хочу вытащить BuildingNodeBase с дочерним тэгом name.

Я пытался посмотреть что бы питон сказал про лямбды. print(help(lambda)) Тогда получаю: print(help(lambda)) ^ SyntaxError: invalid syntax Почему?

Когда я запускаю ‘python setup.py sdist’, то всегда встречаюсь с ошибкой ‘SyntaxError: invalid syntax’. Ниже мой исходный код: printlist.py def printlist(the_list): for eachitem in the_list.

Хочу парсить HTML с lxml с помощью XPath выражения. Моя проблема заключается в совпадении для содержимого тега: Например, дан элемент Example Я могу.

Добрый день, Я разрабатываю скрипт в python и пока пытаюсь его скомпилировать из terminator/terminal i всегда получаю эту ошибку, но не могу понять где ошибка синтаксиса? Файл _case1.py, строка 128.

Вот мой xpath: FSP FSM1/FSP1 FSP .

Я пытаюсь понять и XPath который был отправлен мне для использования с ACORD XML формы (общий формат в страховке). XPath который они мне прислали есть (урезан для краткости).

Am начинаю учить python 3. Am чтение из ebook A byte of Python. Я получил вышеуказанную ошибку при попытке запустить следующую программу: # Filename: backup_ver1.py import os import time source =.

Источник

I am new to Python and very new to web scraping, but I am trying to build a web scraper for this site: https://www.fortune.com/2019/09/23/term-sheet-monday-september-23/

However, my scraper is running into issues before I am even able to get any data from the website. It kicks back: «2019-12-31 09:37:16 [scrapy.core.scraper] ERROR: Spider error processing https://www.fortune.com/2019/09/23/term-sheet-monday-september-23/> (referer: None)».

I created a scraper that is very similar that worked well. I cannot figure out why this is happening on this website.

Any help or suggestions are appreciated!

My spider looks like this:

Import necessary packages

import scrapy import numpy as np import pandas as pd from scrapy.crawler import CrawlerProcess

Define Spider

class Term_Sheet_Spider(scrapy.Spider): name = «Single_Page_Scraper»

    def start_requests(self):

    url = "https://www.fortune.com/2019/09/23/term-sheet-monday-september-23/"
    yield scrapy.Request(url = url, callback = self.parse)

def parse(self,response):
    vc_deals = response.xpath('//*[contains(@target,"_blank")]/text()')
    vc_deals_ext = vc_deals.extract()
    companies = response.css('a[target = "_blank"] > strong::text')
    companies_ext = companies.extract_first()
    dict_vc_2[companies_ext] = vc_deals_ext
    vc_printouts_2.append(companies_ext)

Define Empty Lists

vc_printouts_2 = [] links_list_2 = [] dict_vc_2 = {}

Run Spider

process = CrawlerProcess() process.crawl(Term_Sheet_Spider) process.start()

This is what gets outputted from running the code:

2019-12-31 09:37:15 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: scrapybot) 2019-12-31 09:37:15 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.7.4 (default, Aug 9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.7, Platform Windows-10-10.0.18362-SP0 2019-12-31 09:37:15 [scrapy.crawler] INFO: Overridden settings: {} 2019-12-31 09:37:15 [scrapy.extensions.telnet] INFO: Telnet Password: aab47f83ef3b5653 2019-12-31 09:37:15 [scrapy.middleware] INFO: Enabled extensions: [‘scrapy.extensions.corestats.CoreStats’, ‘scrapy.extensions.telnet.TelnetConsole’, ‘scrapy.extensions.logstats.LogStats’] 2019-12-31 09:37:15 [scrapy.middleware] INFO: Enabled downloader middlewares: [‘scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware’, ‘scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware’, ‘scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware’, ‘scrapy.downloadermiddlewares.useragent.UserAgentMiddleware’, ‘scrapy.downloadermiddlewares.retry.RetryMiddleware’, ‘scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware’, ‘scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware’, ‘scrapy.downloadermiddlewares.redirect.RedirectMiddleware’, ‘scrapy.downloadermiddlewares.cookies.CookiesMiddleware’, ‘scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware’, ‘scrapy.downloadermiddlewares.stats.DownloaderStats’] 2019-12-31 09:37:15 [scrapy.middleware] INFO: Enabled spider middlewares: [‘scrapy.spidermiddlewares.httperror.HttpErrorMiddleware’, ‘scrapy.spidermiddlewares.offsite.OffsiteMiddleware’, ‘scrapy.spidermiddlewares.referer.RefererMiddleware’, ‘scrapy.spidermiddlewares.urllength.UrlLengthMiddleware’, ‘scrapy.spidermiddlewares.depth.DepthMiddleware’] 2019-12-31 09:37:15 [scrapy.middleware] INFO: Enabled item pipelines: [] 2019-12-31 09:37:15 [scrapy.core.engine] INFO: Spider opened 2019-12-31 09:37:15 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2019-12-31 09:37:15 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2019-12-31 09:37:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.fortune.com/2019/09/23/term-sheet-monday-september-23/> (referer: None) 2019-12-31 09:37:16 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.fortune.com/2019/09/23/term-sheet-monday-september-23/> (referer: None) Traceback (most recent call last): File «C:UserskylesAnaconda3libsite-packagesparselselector.py», line 238, in xpath **kwargs) File «src/lxml/etree.pyx», line 1581, in lxml.etree._Element.xpath File «src/lxml/xpath.pxi», line 305, in lxml.etree.XPathElementEvaluator.call File «src/lxml/xpath.pxi», line 225, in lxml.etree._XPathEvaluatorBase._handle_result lxml.etree.XPathEvalError: Invalid expression

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File «C:UserskylesAnaconda3libsite-packagestwistedinternetdefer.py», line 654, in _runCallbacks current.result = callback(current.result, *args, **kw) File «<ipython-input-2-440c0105f261>», line 11, in parse companies = response.xpath(‘//a[target = «_blank» > strong::text’) File «C:UserskylesAnaconda3libsite-packagesscrapyhttpresponsetext.py», line 119, in xpath return self.selector.xpath(query, **kwargs) File «C:UserskylesAnaconda3libsite-packagesparselselector.py», line 242, in xpath six.reraise(ValueError, ValueError(msg), sys.exc_info()[2]) File «C:UserskylesAnaconda3libsite-packagessix.py», line 692, in reraise raise value.with_traceback(tb) File «C:UserskylesAnaconda3libsite-packagesparselselector.py», line 238, in xpath **kwargs) File «src/lxml/etree.pyx», line 1581, in lxml.etree._Element.xpath File «src/lxml/xpath.pxi», line 305, in lxml.etree.XPathElementEvaluator.call File «src/lxml/xpath.pxi», line 225, in lxml.etree._XPathEvaluatorBase._handle_result ValueError: XPath error: Invalid expression in //a[target = «_blank» > strong::text 2019-12-31 09:37:16 [scrapy.core.engine] INFO: Closing spider (finished) 2019-12-31 09:37:16 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {‘downloader/request_bytes’: 256, ‘downloader/request_count’: 1, ‘downloader/request_method_count/GET’: 1, ‘downloader/response_bytes’: 60503, ‘downloader/response_count’: 1, ‘downloader/response_status_count/200’: 1, ‘elapsed_time_seconds’: 0.962091, ‘finish_reason’: ‘finished’, ‘finish_time’: datetime.datetime(2019, 12, 31, 16, 37, 16, 682486), ‘log_count/DEBUG’: 1, ‘log_count/ERROR’: 1, ‘log_count/INFO’: 10, ‘response_received_count’: 1, ‘scheduler/dequeued’: 1, ‘scheduler/dequeued/memory’: 1, ‘scheduler/enqueued’: 1, ‘scheduler/enqueued/memory’: 1, ‘spider_exceptions/ValueError’: 1, ‘start_time’: datetime.datetime(2019, 12, 31, 16, 37, 15, 720395)} 2019-12-31 09:37:16 [scrapy.core.engine] INFO: Spider closed (finished)