Instruction:
Best way to receive the 'return' value from a python generator

Since Python 3.3, if a generator function returns a value, that becomes the value for the StopIteration exception that is raised. This can be collected a number of ways: The value of a yield from expression, which implies the enclosing function is also a generator. Wrapping a call to next() or .send() in a try/except block. However, if I'm simply wanting to iterate over the generator in a for loop - the easiest way - there doesn't appear to be a way to collect the value of the StopIteration exception, and thus the return value. Im using a simple example where the generator yields values, and returns some kind of summary at the end (running totals, averages, timing statistics, etc). for i in produce_values(): do_something(i) values_summary = ....?? One way is to handle the loop myself: values_iter = produce_values() try: while True: i = next(values_iter) do_something(i) except StopIteration as e: values_summary = e.value But this throws away the simplicity of the for loop. I can't use yield from since that requires the calling code to be, itself, a generator. Is there a simpler way than the roll-ones-own for loop shown above?

Output: You can think of the value attribute of StopIteration (and arguably StopIteration itself) as implementation details, not designed to be used in "normal" code. Have a look at PEP 380 that specifies the yield from feature of Python 3.3: It discusses that some alternatives of using StopIteration to carry the return value where considered. Since you are not supposed to get the return value in an ordinary for loop, there is no syntax for it. The same way as you are not supposed to catch the StopIteration explicitly. A nice solution for your situation would be a small utility class (might be useful enough for the standard library): class Generator: def __init__(self, gen): self.gen = gen def __iter__(self): self.value = yield from self.gen return self.value This wraps any generator and catches its return value to be inspected later: >>> def test(): ... yield 1 ... return 2 ... >>> gen = Generator(test()) >>> for i in gen: ... print(i) ... 1 >>> print(gen.value) 2 The line return self.value in __iter__() ensures that the return value will be propagated correctly when Generator instances are nested (e.g. gen = Generator(Generator(test()))).


Instruction:
OSError: [WinError 193] %1 is not a valid Win32 application

I am trying to call a Python file "hello.py" from within the python interpreter with subprocess. But I am unable to resolve this error. [Python 3.4.1]. import subprocess subprocess.call(['hello.py', 'htmlfilename.htm']) Traceback (most recent call last): File "<pyshell#42>", line 1, in <module> subprocess.call(['hello.py', 'htmlfilename.htm']) File "C:\Python34\lib\subprocess.py", line 537, in call with Popen(*popenargs, **kwargs) as p: File "C:\Python34\lib\subprocess.py", line 858, in __init__ restore_signals, start_new_session) File "C:\Python34\lib\subprocess.py", line 1111, in _execute_child startupinfo) OSError: [WinError 193] %1 is not a valid Win32 application Also is there any alternate way to "call a python script with arguments" other than using subprocess?

Output: %1 refers to the first argument of the command, for some reason it was not replaced with hello.py. The file hello.py is not an executable file. You need to specify the executable: subprocess.call(['python.exe', 'hello.py', 'htmlfilename.htm']) You'll need python.exe to be visible on the search path, or you could pass the full path to the executable file that is running the calling script: import sys subprocess.call([sys.executable, 'hello.py', 'htmlfilename.htm'])


Instruction:
How to invoke a function on an object dynamically by name?

In Python, say I have a string that contains the name of a class function that I know a particular object will have, how can I invoke it? That is: obj = MyClass() # this class has a method doStuff() func = "doStuff" # how to call obj.doStuff() using the func variable?

Output: Use the getattr built-in function. See the documentation obj = MyClass() try: func = getattr(obj, "dostuff") func() except AttributeError: print("dostuff not found")


Instruction:
When and how to use Tornado? When is it useless?

Ok, Tornado is non-blocking and quite fast and it can handle a lot of standing requests easily. But I guess it's not a silver bullet and if we just blindly run Django-based or any other site with Tornado it won't give any performance boost. I couldn't find comprehensive explanation of this, so I'm asking it here: When should Tornado be used? When is it useless? When using it, what should be taken into account? How can we make inefficient site using Tornado? There is a server and a webframework. When should we use framework and when can we replace it with other one?

Output: There is a server and a webframework. When should we use framework and when can we replace it with other one? This distinction is a bit blurry. If you are only serving static pages, you would use one of the fast servers like lighthttpd. Otherwise, most servers provide a varying complexity of framework to develop web applications. Tornado is a good web framework. Twisted is even more capable and is considered a good networking framework. It has support for lot of protocols. Tornado and Twisted are frameworks that provide support non-blocking, asynchronous web / networking application development. When should Tornado be used? When is it useless? When using it, what should be taken into account? By its very nature, Async / Non-Blocking I/O works great when it is I/O intensive and not computation intensive. Most web / networking applications suits well for this model. If your application demands certain computational intensive task to be done then it has to be delegated to some other service that can handle it better. While Tornado / Twisted can do the job of web server, responding to web requests. How can we make inefficient site using Tornado? Do any thing computational intensive task Introduce blocking operations But I guess it's not a silver bullet and if we just blindly run Django-based or any other site with Tornado it won't give any performance boost. Performance is usually a characteristic of complete web application architecture. You can bring down the performance with most web frameworks, if the application is not designed properly. Think about caching, load balancing etc. Tornado and Twisted provide reasonable performance and they are good for building performant web applications. You can check out the testimonials for both twisted and tornado to see what they are capable of.


Instruction:
Python - round up to the nearest ten

If I get the number 46 and I want to round up to the nearest ten. How do can I do this in python? 46 goes to 50.

Output: You can use math.ceil() to round up, and then multiply by 10 Python 2 import math def roundup(x): return int(math.ceil(x / 10.0)) * 10 Python 3 (the only difference is you no longer need to cast the result as an int) import math def roundup(x): return math.ceil(x / 10.0) * 10 To use just do >>roundup(45) 50


Instruction:
Remove the last N elements of a list

Is there a a better way to remove the last N elements of a list. for i in range(0,n): lst.pop( )

Output: if you wish to remove the last n elements, in other words, keep first len - n elements: lst = lst[:len(lst)-n] Note: This is not an in memory operation. It would create a shallow copy.


Instruction:
How to get column names from SQLAlchemy result (declarative syntax)

I am working in a pyramid project and I've the table in SQLAlchemy in declarative syntax """models.py""" class Projects(Base): __tablename__ = 'projects' __table_args__ = {'autoload': True} I get the results by using """"views.py""" session = DBSession() row_data = session.query(Projects).filter_by(id=1).one() How can I get the column names from this result. PS: I am unable to use this method since I am using the declarative syntax.

Output: The difference is between ORM and non-ORM, not declarative, which is just a helper for the ORM. Query has an attribute column_descriptions that was added for this purpose: https://docs.sqlalchemy.org/en/20/orm/queryguide/query.html#sqlalchemy.orm.Query.column_descriptions


Instruction:
python enums with attributes

Consider: class Item: def __init__(self, a, b): self.a = a self.b = b class Items: GREEN = Item('a', 'b') BLUE = Item('c', 'd') Is there a way to adapt the ideas for simple enums to this case? (see this question) Ideally, as in Java, I would like to cram it all into one class. Java model: enum EnumWithAttrs { GREEN("a", "b"), BLUE("c", "d"); EnumWithAttrs(String a, String b) { this.a = a; this.b = b; } private String a; private String b; /* accessors and other java noise */ }

Output: Python 3.4 has a new Enum data type (which has been backported as enum34 and enhanced as aenum1). Both enum34 and aenum2 easily support your use case: aenum (Python 2/3) import aenum class EnumWithAttrs(aenum.AutoNumberEnum): _init_ = 'a b' GREEN = 'a', 'b' BLUE = 'c', 'd' enum34 (Python 2/3) or standard library enum (Python 3.4+) import enum class EnumWithAttrs(enum.Enum): def __new__(cls, *args, **kwds): value = len(cls.__members__) + 1 obj = object.__new__(cls) obj._value_ = value return obj def __init__(self, a, b): self.a = a self.b = b GREEN = 'a', 'b' BLUE = 'c', 'd' And in use: >>> EnumWithAttrs.BLUE <EnumWithAttrs.BLUE: 1> >>> EnumWithAttrs.BLUE.a 'c' 1 Disclosure: I am the author of the Python stdlib Enum, the enum34 backport, and the Advanced Enumeration (aenum) library. 2 aenum also supports NamedConstants and metaclass-based NamedTuples.


Instruction:
What difference does it make to use "self" to define a member in a Python class?

How do these 2 classes differ? class A(): x = 3 class B(): def __init__(self): self.x = 3 Is there any significant difference?

Output: A.x is a class variable. B's self.x is an instance variable. i.e. A's x is shared between instances. It would be easier to demonstrate the difference with something that can be modified like a list: class A: x = [] def add(self): self.x.append(1) class B: def __init__(self): self.x = [] def add(self): self.x.append(1) x = A() y = A() x.add() y.add() print("A x:", x.x) x = B() y = B() x.add() y.add() print("B x:", x.x) Output A x: [1, 1] B x: [1]


Instruction:
Remove text between () and []

I have a very long string of text with () and [] in it. I'm trying to remove the characters between the parentheses and brackets but I cannot figure out how. The list is similar to this: x = "This is a sentence. (once a day) [twice a day]" This list isn't what I'm working with but is very similar and a lot shorter.

Output: This should work for parentheses. Regular expressions will "consume" the text it has matched so it won't work for nested parentheses. import re regex = re.compile(".*?\((.*?)\)") result = re.findall(regex, mystring) or this would find one set of parentheses, simply loop to find more: start = mystring.find("(") end = mystring.find(")") if start != -1 and end != -1: result = mystring[start+1:end]


Instruction:
How do I convert a list of dicts to a dict?

How can I convert a list of dicts to a dict? Below is the list of dicts: data = [{'name': 'John Doe', 'age': 37, 'sex': 'M'}, {'name': 'Lisa Simpson', 'age': 17, 'sex': 'F'}, {'name': 'Bill Clinton', 'age': 57, 'sex': 'M'}] I want to convert this to: data = {'John Doe': {'name': 'John Doe', 'age': 37, 'sex': 'M'}, 'Lisa Simpson': {'name': 'Lisa Simpson', 'age': 17, 'sex': 'F'}, 'Bill Clinton': {'name': 'Bill Clinton', 'age': 57, 'sex': 'M'}}

Output: A possible solution using names as the new keys: new_dict = {} for item in data: name = item['name'] new_dict[name] = item With Python 3.x you can also use dict comprehensions for the same approach in a more nice way: new_dict = {item['name']:item for item in data} As suggested in a comment by Paul McGuire, if you don't want the name in the inner dict, you can do: new_dict = {} for item in data: name = item.pop('name') new_dict[name] = item


Instruction:
Split string into strings by length?

Is there a way to take a string that is 4*x characters long, and cut it into 4 strings, each x characters long, without knowing the length of the string? For example: >>>x = "qwertyui" >>>split(x, one, two, three, four) >>>two 'er'

Output: >>> x = "qwertyui" >>> chunks, chunk_size = len(x), len(x)//4 >>> [ x[i:i+chunk_size] for i in range(0, chunks, chunk_size) ] ['qw', 'er', 'ty', 'ui']


Instruction:
How to get a list of all indexes in python-elasticsearch

How would I get a list of the names of an index in Python? Here is what I have so far: >>> es=e.es >>> es <Elasticsearch([{'host': '14555f777d8097.us-east-1.aws.found.io', 'port': 9200}])> >>> es.indices <elasticsearch.client.indices.IndicesClient object at 0x10de86790> # how to get a list of all indexes in this cluster?

Output: This question comes up when searching for information on retrieving aliases using the python-elasticsearch library. The accepted answer says to use get_aliases but that method has been removed (as of 2017). To get aliases, you can use the following: es.indices.get_alias("*") UPDATE The latest usage should be with a keyword arg: es.indices.get_alias(index="*")


Instruction:
Factorial in numpy and scipy

How can I import factorial function from numpy and scipy separately in order to see which one is faster? I already imported factorial from python itself by import math. But, it does not work for numpy and scipy.

Output: You can import them like this: In [7]: import scipy, numpy, math In [8]: scipy.math.factorial, numpy.math.factorial, math.factorial Out[8]: (<function math.factorial>, <function math.factorial>, <function math.factorial>) scipy.math.factorial and numpy.math.factorial seem to simply be aliases/references for/to math.factorial, that is scipy.math.factorial is math.factorial and numpy.math.factorial is math.factorial should both give True.


Instruction:
Ignore divide by 0 warning in NumPy

I have a function for statistic issues: import numpy as np from scipy.special import gamma as Gamma def Foo(xdata): ... return x1 * ( ( #R is a numpy vector ( ((R - x2)/beta) ** (x3 -1) ) * ( np.exp( - ((R - x2) / x4) ) ) / ( x4 * Gamma(x3)) ).real ) Sometimes I get from the shell the following warning: RuntimeWarning: divide by zero encountered in... I use the numpy isinf function to correct the results of the function in other files, so I do not need this warning. Is there a way to ignore the message? In other words, I do not want the shell to print this message. I do not want to disable all python warnings, just this one.

Output: You can disable the warning with numpy.seterr. Put this before the possible division by zero: np.seterr(divide='ignore') That'll disable zero division warnings globally. If you just want to disable them for a little bit, you can use numpy.errstate in a with clause: with np.errstate(divide='ignore'): # some code here For a zero by zero division (undetermined, results in a NaN), the error behaviour has changed with numpy version 1.8.0: this is now considered "invalid", while previously it was "divide". Thus, if there is a chance you your numerator could be zero as well, use np.seterr(divide='ignore', invalid='ignore') or with np.errstate(divide='ignore', invalid='ignore'): # some code here See the "Compatibility" section in the release notes, last paragraph before the "New Features" section: Comparing NaN floating point numbers now raises the invalid runtime warning. If a NaN is expected the warning can be ignored using np.errstate.


Instruction:
Get value at list/array index or "None" if out of range in Python

Is there clean way to get the value at a list index or None if the index is out or range in Python? The obvious way to do it would be this: if len(the_list) > i: return the_list[i] else: return None However, the verbosity reduces code readability. Is there a clean, simple, one-liner that can be used instead?

Output: Try: try: return the_list[i] except IndexError: return None Or, one liner: l[i] if i < len(l) else None Example: >>> l=list(range(5)) >>> i=6 >>> print(l[i] if i < len(l) else None) None >>> i=2 >>> print(l[i] if i < len(l) else None) 2


Instruction:
Python Array with String Indices

Is it possible to use strings as indices in an array in python? For example: myArray = [] myArray["john"] = "johns value" myArray["jeff"] = "jeffs value" print myArray["john"]

Output: What you want is called an associative array. In python these are called dictionaries. Dictionaries are sometimes found in other languages as “associative memories” or “associative arrays”. Unlike sequences, which are indexed by a range of numbers, dictionaries are indexed by keys, which can be any immutable type; strings and numbers can always be keys. myDict = {} myDict["john"] = "johns value" myDict["jeff"] = "jeffs value" Alternative way to create the above dict: myDict = {"john": "johns value", "jeff": "jeffs value"} Accessing values: print(myDict["jeff"]) # => "jeffs value" Getting the keys (in Python v2): print(myDict.keys()) # => ["john", "jeff"] In Python 3, you'll get a dict_keys, which is a view and a bit more efficient (see views docs and PEP 3106 for details). print(myDict.keys()) # => dict_keys(['john', 'jeff']) If you want to learn about python dictionary internals, I recommend this ~25 min video presentation: https://www.youtube.com/watch?v=C4Kc8xzcA68. It's called the "The Mighty Dictionary".


Instruction:
How do I make a fixed size formatted string in python?

I want to create a formatted string with fixed size with fixed position between fields. An example explains better, here there are clearly 3 distinct fields and the string is a fixed size: XXX 123 98.00 YYYYY 3 1.00 ZZ 42 123.34 How can I apply such formatting to a string in python (2.7)?

Output: Sure, use the .format method. E.g., print('{:10s} {:3d} {:7.2f}'.format('xxx', 123, 98)) print('{:10s} {:3d} {:7.2f}'.format('yyyy', 3, 1.0)) print('{:10s} {:3d} {:7.2f}'.format('zz', 42, 123.34)) will print xxx 123 98.00 yyyy 3 1.00 zz 42 123.34 You can adjust the field sizes as desired. Note that .format works independently of print to format a string. I just used print to display the strings. Brief explanation: 10s format a string with 10 spaces, left justified by default 3d format an integer reserving 3 spaces, right justified by default 7.2f format a float, reserving 7 spaces, 2 after the decimal point, right justfied by default. There are many additional options to position/format strings (padding, left/right justify etc), String Formatting Operations will provide more information. Update for f-string mode. E.g., text, number, other_number = 'xxx', 123, 98 print(f'{text:10} {number:3d} {other_number:7.2f}') For right alignment print(f'{text:>10} {number:3d} {other_number:7.2f}')


Instruction:
Passing multiple parameters to pool.map() function in Python

I need some way to use a function within pool.map() that accepts more than one parameter. As per my understanding, the target function of pool.map() can only have one iterable as a parameter but is there a way that I can pass other parameters in as well? In this case, I need to pass in a few configuration variables, like my Lock() and logging information to the target function. I have tried to do some research and I think that I may be able to use partial functions to get it to work? However I don't fully understand how these work. Any help would be greatly appreciated! Here is a simple example of what I want to do: def target(items, lock): for item in items: # Do cool stuff if (... some condition here ...): lock.acquire() # Write to stdout or logfile, etc. lock.release() def main(): iterable = [1, 2, 3, 4, 5] pool = multiprocessing.Pool() pool.map(target(PASS PARAMS HERE), iterable) pool.close() pool.join()

Output: You can use functools.partial for this (as you suspected): from functools import partial def target(lock, iterable_item): for item in iterable_item: # Do cool stuff if (... some condition here ...): lock.acquire() # Write to stdout or logfile, etc. lock.release() def main(): iterable = [1, 2, 3, 4, 5] pool = multiprocessing.Pool() l = multiprocessing.Lock() func = partial(target, l) pool.map(func, iterable) pool.close() pool.join() Example: def f(a, b, c): print("{} {} {}".format(a, b, c)) def main(): iterable = [1, 2, 3, 4, 5] pool = multiprocessing.Pool() a = "hi" b = "there" func = partial(f, a, b) pool.map(func, iterable) pool.close() pool.join() if __name__ == "__main__": main() Output: hi there 1 hi there 2 hi there 3 hi there 4 hi there 5


Instruction:
Import Python Script Into Another?

I'm going through Zed Shaw's Learn Python The Hard Way and I'm on lesson 26. In this lesson we have to fix some code, and the code calls functions from another script. He says that we don't have to import them to pass the test, but I'm curious as to how we would do so. Link to the lesson | Link to the code to correct And here are the particular lines of code that call on a previous script: words = ex25.break_words(sentence) sorted_words = ex25.sort_words(words) print_first_word(words) print_last_word(words) print_first_word(sorted_words) print_last_word(sorted_words) sorted_words = ex25.sort_sentence(sentence) print sorted_words print_first_and_last(sentence) print_first_a_last_sorted(sentence) Code to Correct: This is the code from the course, that's being referenced Do not edit the question to correct the code def break_words(stuff): """This function will break up words for us.""" words = stuff.split(' ') return words def sort_words(words): """Sorts the words.""" return sorted(words) def print_first_word(words) """Prints the first word after popping it off.""" word = words.poop(0) print word def print_last_word(words): """Prints the last word after popping it off.""" word = words.pop(-1 print word def sort_sentence(sentence): """Takes in a full sentence and returns the sorted words.""" words = break_words(sentence) return sort_words(words) def print_first_and_last(sentence): """Prints the first and last words of the sentence.""" words = break_words(sentence) print_first_word(words) print_last_word(words) def print_first_and_last_sorted(sentence): """Sorts the words then prints the first and last one.""" words = sort_sentence(sentence) print_first_word(words) print_last_word(words) print "Let's practice everything." print 'You\'d need to know \'bout escapes with \\ that do \n newlines and \t tabs.' poem = """ \tThe lovely world with logic so firmly planted cannot discern \n the needs of love nor comprehend passion from intuition and requires an explantion \n\t\twhere there is none. """ print "--------------" print poem print "--------------" five = 10 - 2 + 3 - 5 print "This should be five: %s" % five def secret_formula(started): jelly_beans = started * 500 jars = jelly_beans \ 1000 crates = jars / 100 return jelly_beans, jars, crates start_point = 10000 beans, jars, crates == secret_formula(start-point) print "With a starting point of: %d" % start_point print "We'd have %d jeans, %d jars, and %d crates." % (beans, jars, crates) start_point = start_point / 10 print "We can also do that this way:" print "We'd have %d beans, %d jars, and %d crabapples." % secret_formula(start_pont sentence = "All god\tthings come to those who weight." words = ex25.break_words(sentence) sorted_words = ex25.sort_words(words) print_first_word(words) print_last_word(words) .print_first_word(sorted_words) print_last_word(sorted_words) sorted_words = ex25.sort_sentence(sentence) prin sorted_words print_irst_and_last(sentence) print_first_a_last_sorted(senence)

Output: It depends on how the code in the first file is structured. If it's just a bunch of functions, like: # first.py def foo(): print("foo") def bar(): print("bar") Then you could import it and use the functions as follows: # second.py import first first.foo() # prints "foo" first.bar() # prints "bar" or # second.py from first import foo, bar foo() # prints "foo" bar() # prints "bar" or, to import all the names defined in first.py: # second.py from first import * foo() # prints "foo" bar() # prints "bar" Note: This assumes the two files are in the same directory. It gets a bit more complicated when you want to import names (functions, classes, etc) from modules in other directories or packages.


Instruction:
Assigning a function to a variable

Let's say I have a function def x(): print(20) Now I want to assign the function to a variable called y, so that if I use the y it calls the function x again. if i simply do the assignment y = x(), it returns None.

Output: You simply don't call the function. >>> def x(): >>> print(20) >>> y = x >>> y() 20 The brackets tell Python that you are calling the function, so when you put them there, it calls the function and assigns y the value returned by x (which in this case is None).


Instruction:
How can I tell which python implementation I'm using?

Python has a few different implementations: CPython, Jython, PyPy, etc. I want to programmatically determine which implementation my code is running on. How can I do that? To be specific, I'm looking for a function, say, get_implementation_name(), that could be used like so: impl_name = get_implementation_name() if impl_name == "CPython": print "I can abuse CPython implementation details. (I'm a bad, bad man.)" elif impl_name == "PyPy": print "Can't count on reference-counting garbage collection here..." else: print "I better be careful..."

Output: In [50]: import platform In [52]: platform.python_implementation() Out[52]: 'CPython' Docs: platform.python_implementation


Instruction:
type hinting within a class

class Node: def append_child(self, node: Node): if node != None: self.first_child = node self.child_nodes += [node] How do I do node: Node? Because when I run it, it says name 'Node' is not defined. Should I just remove the : Node and instance check it inside the function? But then how could I access node's properties (which I would expect to be instance of Node class)? I don't know how implement type casting in Python, BTW.

Output: "self" references in type checking are typically done using strings: class Node: def append_child(self, node: 'Node'): if node != None: self.first_child = node self.child_nodes += [node] This is described in the "Forward references" section of PEP-0484. Please note that this doesn't do any type-checking or casting. This is a type hint which python (normally) disregards completely1. However, third party tools (e.g. mypy), use type hints to do static analysis on your code and can generate errors before runtime. Also, starting with python3.7, you can implicitly convert all of your type-hints to strings within a module by using the from __future__ import annotations (and in python4.0, this will be the default). 1The hints are introspectable -- So you could use them to build some kind of runtime checker using decorators or the like if you really wanted to, but python doesn't do this by default.


Instruction:
ImportError: cannot import name '...' from 'collections' using Python 3.10

I am trying to run my program which uses various dependencies, but since upgrading to Python 3.10 this does not work anymore. When I run "python3" in the terminal and from there import my dependencies I get an error: ImportError: cannot import name 'Mapping' from 'collections' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py) This seems to be a general problem, but here is the traceback of my specific case: Traceback (most recent call last): File "/Users/mk/Flasktut/app.py", line 2, in <module> from flask import Flask, render_template File "/Users/mk/Flasktut/env/lib/python3.10/site-packages/flask/__init__.py", line 14, in <module> from jinja2 import escape File "/Users/mk/Flasktut/env/lib/python3.10/site-packages/jinja2/__init__.py", line 33, in <module> from jinja2.environment import Environment, Template File "/Users/mk/Flasktut/env/lib/python3.10/site-packages/jinja2/environment.py", line 16, in <module> from jinja2.defaults import BLOCK_START_STRING, \ File "/Users/mk/Flasktut/env/lib/python3.10/site-packages/jinja2/defaults.py", line 32, in <module> from jinja2.tests import TESTS as DEFAULT_TESTS File "/Users/mk/Flasktut/env/lib/python3.10/site-packages/jinja2/tests.py", line 13, in <module> from collections import Mapping ImportError: cannot import name 'Mapping' from 'collections' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)

Output: Change: from collections import Mapping to from collections.abc import Mapping


Instruction:
socket.error:[errno 99] cannot assign requested address and namespace in python

My server software says errno99: cannot assign requested address while using an ip address other than 127.0.0.1 for binding. But if the IP address is 127.0.0.1 it works. Is it related to namespaces? I am executing my server and client codes in another python program by calling execfile(). I am actually editing the mininet source code.I edited net.py and inside that I used execfile('server.py') execfile('client1.py') and execfile('client2.py').So as soon as "sudo mn --topo single,3" is called along with the creation of 3 hosts my server and client codes will get executed.I have given my server and client codes below. #server code import select import socket import sys backlog = 5 size = 1024 server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server.bind(("10.0.0.1",9999)) server.listen(backlog) input = [server] running = 1 while running: inputready,outputready,exceptready = select.select(input,[],[]) for s in inputready: if s == server: client, address = server.accept() input.append(client) else: l = s.recv(1024) sys.stdout.write(l) server.close() #client code import socket import select import sys import time while(1) : s,addr=server1.accept() data=int(s.recv(4)) s = socket.socket() s.connect(("10.0.0.1",9999)) while (1): f=open ("hello1.txt", "rb") l = f.read(1024) s.send(l) l = f.read(1024) time.sleep(5) s.close()

Output: Stripping things down to basics this is what you would want to test with: import socket server = socket.socket() server.bind(("10.0.0.1", 6677)) server.listen(4) client_socket, client_address = server.accept() print(client_address, "has connected") while True: recvieved_data = client_socket.recv(1024) print(recvieved_data) This works assuming a few things: Your local IP address (on the server) is 10.0.0.1 (This video shows you how) No other software is listening on port 6677 Also note the basic concept of IP addresses: Try the following, open the start menu, in the "search" field type cmd and press enter. Once the black console opens up type ping www.google.com and this should give you and IP address for google. This address is googles local IP and they bind to that and obviously you can not bind to an IP address owned by google. With that in mind, you own your own set of IP addresses. First you have the local IP of the server, but then you have the local IP of your house. In the below picture 192.168.1.50 is the local IP of the server which you can bind to. You still own 83.55.102.40 but the problem is that it's owned by the Router and not your server. So even if you visit http://whatsmyip.com and that tells you that your IP is 83.55.102.40 that is not the case because it can only see where you're coming from.. and you're accessing your internet from a router. In order for your friends to access your server (which is bound to 192.168.1.50) you need to forward port 6677 to 192.168.1.50 and this is done in your router. Assuming you are behind one. If you're in school there's other dilemmas and routers in the way most likely.


Instruction:
What's the mathematical reason behind Python choosing to round integer division toward negative infinity?

I know Python // rounds towards negative infinity and in C++ / is truncating, rounding towards 0. And here's what I know so far: |remainder| -12 / 10 = -1, - 2 // C++ -12 // 10 = -2, + 8 # Python 12 / -10 = -1, 2 // C++ 12 // -10 = -2, - 8 # Python 12 / 10 = 1, 2 // Both 12 // 10 = 1, 2 -12 / -10 = 1, - 2 // Both = 2, + 8 C++: 1. m%(-n) == m%n 2. -m%n == -(m%n) 3. (m/n)*n + m%n == m Python: 1. m%(-n) == -8 == -(-m%n) 2. (m//n)*n + m%n == m But why Python // choose to round towards negative infinity? I didn't find any resources explain that, but only find and hear people say vaguely: "for mathematics reasons". For example, in Why is -1/2 evaluated to 0 in C++, but -1 in Python?: People dealing with these things in the abstract tend to feel that rounding toward negative infinity makes more sense (that means it's compatible with the modulo function as defined in mathematics, rather than % having a somewhat funny meaning). But I don't see C++ 's / not being compatible with the modulo function. In C++, (m/n)*n + m%n == m also applies. So what's the (mathematical) reason behind Python choosing rounding towards negative infinity? See also Guido van Rossum's old blog post on the topic.

Output: But why Python // choose to round towards negative infinity? I'm not sure if the reason why this choice was originally made is documented anywhere (although, for all I know, it could be explained in great length in some PEP somewhere), but we can certainly come up with various reasons why it makes sense. One reason is simply that rounding towards negative (or positive!) infinity means that all numbers get rounded the same way, whereas rounding towards zero makes zero special. The mathematical way of saying this is that rounding down towards −∞ is translation invariant, i.e. it satisfies the equation: round_down(x + k) == round_down(x) + k for all real numbers x and all integers k. Rounding towards zero does not, since, for example: round_to_zero(0.5 - 1) != round_to_zero(0.5) - 1 Of course, other arguments exist too, such as the argument you quote based on compatibility with (how we would like) the % operator (to behave) — more on that below. Indeed, I would say the real question here is why Python's int() function is not defined to round floating point arguments towards negative infinity, so that m // n would equal int(m / n). (I suspect "historical reasons".) Then again, it's not that big of a deal, since Python does at least have math.floor() that does satisfy m // n == math.floor(m / n). But I don't see C++ 's / not being compatible with the modulo function. In C++, (m/n)*n + m%n == m also applies. True, but retaining that identity while having / round towards zero requires defining % in an awkward way for negative numbers. In particular, we lose both of the following useful mathematical properties of Python's %: 0 <= m % n < n for all m and all positive n; and (m + k * n) % n == m % n for all integers m, n and k. These properties are useful because one of the main uses of % is "wrapping around" a number m to a limited range of length n. For example, let's say we're trying to calculate directions: let's say heading is our current compass heading in degrees (counted clockwise from due north, with 0 <= heading < 360) and that we want to calculate our new heading after turning angle degrees (where angle > 0 if we turn clockwise, or angle < 0 if we turn counterclockwise). Using Python's % operator, we can calculate our new heading simply as: heading = (heading + angle) % 360 and this will simply work in all cases. However, if we try to to use this formula in C++, with its different rounding rules and correspondingly different % operator, we'll find that the wrap-around doesn't always work as expected! For example, if we start facing northwest (heading = 315) and turn 90° clockwise (angle = 90), we'll indeed end up facing northeast (heading = 45). But if then try to turn back 90° counterclockwise (angle = -90), with C++'s % operator we won't end up back at heading = 315 as expected, but instead at heading = -45! To get the correct wrap-around behavior using the C++ % operator, we'll instead need to write the formula as something like: heading = (heading + angle) % 360; if (heading < 0) heading += 360; or as: heading = ((heading + angle) % 360) + 360) % 360; (The simpler formula heading = (heading + angle + 360) % 360 will only work if we can always guarantee that heading + angle >= -360.) This is the price you pay for having a non-translation-invariant rounding rule for division, and consequently a non-translation-invariant % operator.


Instruction:
How to transform an XML file using XSLT in Python?

Good day! Need to convert xml using xslt in Python. I have a sample code in php. How to implement this in Python or where to find something similar? Thank you! $xmlFileName = dirname(__FILE__)."example.fb2"; $xml = new DOMDocument(); $xml->load($xmlFileName); $xslFileName = dirname(__FILE__)."example.xsl"; $xsl = new DOMDocument; $xsl->load($xslFileName); // Configure the transformer $proc = new XSLTProcessor(); $proc->importStyleSheet($xsl); // attach the xsl rules echo $proc->transformToXML($xml);

Output: Using lxml, import lxml.etree as ET dom = ET.parse(xml_filename) xslt = ET.parse(xsl_filename) transform = ET.XSLT(xslt) newdom = transform(dom) print(ET.tostring(newdom, pretty_print=True))


Instruction:
no module named zlib

First, please bear with me. I have hard time telling others my problem and this is a long thread... I am using pythonbrew to run multiple versions of python in Ubuntu 10.10. For installing pythonbrew and how it works, please refers to this link below http://www.howopensource.com/2011/05/how-to-install-and-manage-different-versions-of-python-in-linux/ After reading a couple stackoverflow threads, I finally found the file called Setup under this directory: ~/.pythonbrew/pythons/Python-2.7.1/lib/python2.7/config In this Setup file I see # Andrew Kuchling's zlib module. # This require zlib 1.1.3 (or later). # See http://www.gzip.org/zlib/ # zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz I uncommented the last line, then I ran python -v again. However, I received the same error when I tried import zlib, so I guess I have to do something to install zlib into the lib. But I am clueless about what I need to do. Can someone please direct me in the right direction??? Thank you very much! I am doing this because I want to use different version of python in different virtualenv I created. When I did virtualenv -p python2.7 I received no module named zlib. jwxie518@jwxie518-P5E-VM-DO:~$ virtualenv -p python2.7 --no-site-packages testenv Running virtualenv with interpreter /home/jwxie518/.pythonbrew/pythons/Python-2.7.1/bin/python2.7 Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/virtualenv.py", line 17, in <module> import zlib ImportError: No module named zlib EDIT I have to install 2.7.1 by appending --force. I am developing Django, and I need some of these missing modules, for example sqlite3, and to create my virtualenv I definitely need zlib. If I just use the system default (2.6.6), I have no problem. To do this with system default, all I need to do is virtualenv --no-site-packages testenv Thanks! (2nd edit) I installed 3.2 also and I tested it without problem, so I guess my problem comes down to how to install the missing module(s). jwxie518@jwxie518-P5E-VM-DO:~$ virtualenv -p python3.2 testenv Running virtualenv with interpreter /home/jwxie518/.pythonbrew/pythons/Python-3.2/bin/python3.2 New python executable in testenv/bin/python3.2 Also creating executable in testenv/bin/python Installing distribute..................................................................................................................................................................................................................................................................................................................................done. Installing pip...............done. jwxie518@jwxie518-P5E-VM-DO:~$ virtualenv -p python3.2 --no-site-packages testenv Running virtualenv with interpreter /home/jwxie518/.pythonbrew/pythons/Python-3.2/bin/python3.2 New python executable in testenv/bin/python3.2 Also creating executable in testenv/bin/python Installing distribute..................................................................................................................................................................................................................................................................................................................................done. Installing pip...............done.

Output: Sounds like you need to install the devel package for zlib, probably want to do something like # ubuntu 12,14,16,18,20.04+ sudo apt-get install zlib1g-dev Instead of using python-brew you might want to consider just compiling by hand, it's not very hard. Just download the source, and configure, make, make install. You'll want to at least set --prefix to somewhere, so it'll get installed where you want. ./configure --prefix=/opt/python2.7 + other options make make install You can check what configuration options are available with ./configure --help and see what your system python was compiled with by doing: python -c "import sysconfig; print sysconfig.get_config_var('CONFIG_ARGS')" The key is to make sure you have the development packages installed for your system, so that Python will be able to build the zlib, sqlite3, etc modules. The python docs cover the build process in more detail: http://docs.python.org/using/unix.html#building-python.


Instruction:
Python 3 bytes formatting

In Python 3, one can format a string like: "{0}, {1}, {2}".format(1, 2, 3) But how to format bytes? b"{0}, {1}, {2}".format(1, 2, 3) raises AttributeError: 'bytes' object has no attribute 'format'. If there is no format method for bytes, how to do the formatting or "rewriting" of bytes?

Output: As of Python 3.5, % formatting will work for bytes, too! This was part of PEP 461, authored by Ethan Furman: PEP: 461 Title: Adding % formatting to bytes and bytearray Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman <ethan at stoneleaf.us> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25, 2014-03-27 Resolution: Abstract ======== This PEP proposes adding % formatting operations similar to Python 2's ``str`` type to ``bytes`` and ``bytearray`` [1]_ [2]_. Rationale ========= While interpolation is usually thought of as a string operation, there are cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. Motivation ========== With Python 3 and the split between ``str`` and ``bytes``, one small but important area of programming became slightly more difficult, and much more painful -- wire format protocols [3]_. This area of programming is characterized by a mixture of binary data and ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in writing new wire format code, and in porting Python 2 wire format code. Common use-cases include ``dbf`` and ``pdf`` file formats, ``email`` formats, and ``FTP`` and ``HTTP`` communications, among many others. PEP 461 was accepted by Guido van Rossum on March 27, 2014: Accepted. Congrats with marshalling yet another quite contentious discussion, and putting up with my last-minute block-headedness! From this, we can obviously conclude that % is no longer scheduled for deprecation (as was announced with Python 3.1).


Instruction:
Adding element to list while iterating

I know that it is not allowed to remove elements while iterating a list, but is it allowed to add elements to a python list while iterating. Here is an example: for a in myarr: if somecond(a): myarr.append(newObj()) I have tried this in my code and it seems to work fine, however I don't know if it's because I am just lucky and that it will break at some point in the future? I prefer not to copy the list since myarr is huge, and therefore it would be too slow. Also I need to check the appended objects with somecond(). At some point somecond(a) will be false, so there can not be an infinite loop. Each object in myarr has a size, and each time somecond(a) is true and a new object is appended to the list, the new object will have a size smaller than a. somecond() has an epsilon for how small objects can be and if they are too small it will return "false".

Output: You could use the islice from itertools to create an iterator over a smaller portion of the list. Then you can append entries to the list without impacting the items you're iterating over: islice(myarr, 0, len(myarr)-1) Even better, you don't even have to iterate over all the elements. You can increment a step size.


Instruction:
auto.arima() equivalent for python

I am trying to predict weekly sales using ARMA ARIMA models. I could not find a function for tuning the order(p,d,q) in statsmodels. Currently R has a function forecast::auto.arima() which will tune the (p,d,q) parameters. How do I go about choosing the right order for my model? Are there any libraries available in python for this purpose?

Output: You can implement a number of approaches: ARIMAResults include aic and bic. By their definition, (see here and here), these criteria penalize for the number of parameters in the model. So you may use these numbers to compare the models. Also scipy has optimize.brute which does grid search on the specified parameters space. So a workflow like this should work: def objfunc(order, exog, endog): from statsmodels.tsa.arima.model import ARIMA fit = ARIMA(endog, order, exog).fit() return fit.aic() from scipy.optimize import brute grid = (slice(1, 3, 1), slice(1, 3, 1), slice(1, 3, 1)) brute(objfunc, grid, args=(exog, endog), finish=None) Make sure you call brute with finish=None. You may obtain pvalues from ARIMAResults. So a sort of step-forward algorithm is easy to implement where the degree of the model is increased across the dimension which obtains lowest p-value for the added parameter. Use ARIMAResults.predict to cross-validate alternative models. The best approach would be to keep the tail of the time series (say most recent 5% of data) out of sample, and use these points to obtain the test error of the fitted models.


Instruction:
How to combine python asyncio with threads?

I have successfully built a RESTful microservice with Python asyncio and aiohttp that listens to a POST event to collect realtime events from various feeders. It then builds an in-memory structure to cache the last 24h of events in a nested defaultdict/deque structure. Now I would like to periodically checkpoint that structure to disc, preferably using pickle. Since the memory structure can be >100MB I would like to avoid holding up my incoming event processing for the time it takes to checkpoint the structure. I'd rather create a snapshot copy (e.g. deepcopy) of the structure and then take my time to write it to disk and repeat on a preset time interval. I have been searching for examples on how to combine threads (and is a thread even the best solution for this?) and asyncio for that purpose but could not find something that would help me. Any pointers to get started are much appreciated!

Output: It's pretty simple to delegate a method to a thread or sub-process using loop.run_in_executor: import asyncio import time from concurrent.futures import ProcessPoolExecutor def cpu_bound_operation(x): time.sleep(x) # This is some operation that is CPU-bound async def main(): loop = asyncio.get_running_loop() # Run cpu_bound_operation in the ProcessPoolExecutor # This will make your coroutine block, but won't block # the event loop; other coroutines can run in meantime. await loop.run_in_executor(p, cpu_bound_operation, 5) p = ProcessPoolExecutor(2) # Create a Process Pool with 2 processes asyncio.run(main()) As for whether to use a ProcessPoolExecutor or ThreadPoolExecutor, that's kind of hard to say; pickling a large object will definitely eat some CPU cycles, which initially would make you think ProcessPoolExecutor is the way to go. However, passing your 100MB object to a Process in the pool would require pickling the instance in your main process, sending the bytes to the child process via IPC, unpickling it in the child, and then pickling it again so you can write it to disk. Given that, my guess is the pickling/unpickling overhead will be large enough that you're better off using a ThreadPoolExecutor, even though you're going to take a performance hit because of the GIL. That said, it's very simple to test both ways and find out for sure, so you might as well do that.


Instruction:
multiprocessing: map vs map_async

What's the difference between using map and map_async? Are they not running the same function after distributing the items from the list to 4 processes? So is it wrong to presume both are running asynchronous and parallel? def f(x): return 2*x p=Pool(4) l=[1,2,3,4] out1=p.map(f,l) #vs out2=p.map_async(f,l)

Output: There are four choices to mapping jobs to processes. You have to consider multi-args, concurrency, blocking, and ordering. map and map_async only differ with respect to blocking. map_async is non-blocking where as map is blocking So let's say you had a function from multiprocessing import Pool import time def f(x): print(x*x) if __name__ == '__main__': pool = Pool(processes=4) pool.map(f, range(10)) r = pool.map_async(f, range(10)) # DO STUFF print('HERE') print('MORE') r.wait() print('DONE') Example output: 0 1 4 9 16 25 36 49 64 81 0 HERE 1 4 MORE 16 25 36 9 49 64 81 DONE pool.map(f, range(10)) will wait for all 10 of those function calls to finish so we see all the prints in a row. r = pool.map_async(f, range(10)) will execute them asynchronously and only block when r.wait() is called so we see HERE and MORE in between but DONE will always be at the end.


Instruction:
Use the default Python rather than the Anaconda installation when called from the terminal

I recently installed the Anaconda version of Python. Now when I type python into the terminal it opens the Anaconda distribution rather than the default distribution. How do I get it to use the default version for the command python on Linux (Ubuntu 12.04 (Precise Pangolin))?

Output: Anaconda adds the path to your .bashrc, so it is found first. You can add the path to your default Python instance to .bashrc or remove the path to Anaconda if you don't want to use it. You can also use the full path /usr/bin/python in Bash to use the default Python interpreter. If you leave your .bashrc file as is, any command you run using python will use the Anaconda interpreter. If you want, you could also use an alias for each interpreter. You will see something like export PATH=$HOME/anaconda/bin:$PATH in your .bashrc file. So basically, if you want to use Anaconda as your main everyday interpreter, use the full path to your default Python or create an alias. If you want it the other way around, remove the export PATH=.... from bashrc and use full path to Anaconda Python interpreter.


Instruction:
Can sklearn random forest directly handle categorical features?

Say I have a categorical feature, color, which takes the values ['red', 'blue', 'green', 'orange'], and I want to use it to predict something in a random forest. If I one-hot encode it (i.e. I change it to four dummy variables), how do I tell sklearn that the four dummy variables are really one variable? Specifically, when sklearn is randomly selecting features to use at different nodes, it should either include the red, blue, green and orange dummies together, or it shouldn't include any of them. I've heard that there's no way to do this, but I'd imagine there must be a way to deal with categorical variables without arbitrarily coding them as numbers or something like that.

Output: No, there isn't. Somebody's working on this and the patch might be merged into mainline some day, but right now there's no support for categorical variables in scikit-learn except dummy (one-hot) encoding.


Instruction:
How to specify version in only one place when using pyproject.toml?

My package version is defined in two places: __version__ = 1.2.3 in mypackage/__init__.py version = "1.2.3" in pyproject.toml (I am using Poetry) I have to update both whenever I bump the version which is annoying and not DRY. Is there a way to make Python read the version from the TOML, or to make the TOML read the version from Python?

Output: After you have installed your project - either in editable mode by poetry install or from the wheel - you can access several metadata via importlib.metadata (importlib_metadata for python < 3.8). So keep the version only in the pyproject.toml and use this in your python code: import importlib.metadata __version__ = importlib.metadata.version("mypackage")


Instruction:
How to get row number in dataframe in Pandas?

How can I get the number of the row in a dataframe that contains a certain value in a certain column using Pandas? For example, I have the following dataframe: ClientID LastName 0 34 Johnson 1 67 Smith 2 53 Brows How can I find the number of the row that has 'Smith' in 'LastName' column?

Output: Note that a dataframe's index could be out of order, or not even numerical at all. If you don't want to use the current index and instead renumber the rows sequentially, then you can use df.reset_index() together with the suggestions below To get all indices that matches 'Smith' >>> df[df['LastName'] == 'Smith'].index Int64Index([1], dtype='int64') or as a numpy array >>> df[df['LastName'] == 'Smith'].index.to_numpy() # .values on older versions array([1]) or if there is only one and you want the integer, you can subset >>> df[df['LastName'] == 'Smith'].index[0] 1 You could use the same boolean expressions with .loc, but it is not needed unless you also want to select a certain column, which is redundant when you only want the row number/index.


Instruction:
String concatenation without '+' operator

I was playing with python and I realized we don't need to use '+' operator to concatenate static strings. But it fails if I assign it to a variable. For example: string1 = 'Hello' 'World' #1 works fine string2 = 'Hello' + 'World' #2 also works fine string3 = 'Hello' string4 = 'World' string5 = string3 string4 #3 causes syntax error string6 = string3 + string4 #4 works fine Now I have two questions: Why statement 3 does not work while statement 1 does? Is there any technical difference such as calculation speed etc. between statement 1 and 2?

Output: From the Python 3.12 docs: Multiple adjacent string literals (delimited by whitespace), possibly using different quoting conventions, are allowed, and their meaning is the same as their concatenation. Thus, "hello" 'world' is equivalent to "helloworld". Statement 3 doesn't work because: The ‘+’ operator must be used to concatenate string expressions at run time. Notice that the title of the subheader in the docs is "string literal concatenation" too. This only works for string literals, not other objects. There's probably no difference. If there is, it's probably extremely tiny and nothing that anyone should worry about. Also, understand that there can be dangers to this: >>> def foo(bar, baz=None): ... return bar ... >>> foo("bob" ... "bill") 'bobbill' This is a perfect example of where Errors should never pass silently. What if I wanted "bill" to be the argument baz? I have forgotton a comma, but no error is raised. Instead, concatenation has taken place.


Instruction:
Is there any disadvantage in using PYTHONDONTWRITEBYTECODE in Docker?

In many Docker tutorials based on Python (such as: this one) they use the option PYTHONDONTWRITEBYTECODE in order to make Python avoid to write .pyc files on the import of source modules (This is equivalent to specifying the -B option). What are the risks and advantages of setting this option up?

Output: When you run a single python process in the container, which does not spawn other python processes itself during its lifetime, then there is no "risk" in doing that. Storing byte code on disk is used to compile python into byte code just upon the first invocation of a program and its dependent libraries to save that step upon the following invocations. In a container the process runs just once, therefore setting this option makes sense.


Instruction:
How to fix "ImportError: DLL load failed" while importing win32api

I'm setting up an autoclicker in Python 3.8 and I need win32api for GetAsyncKeyState but it always gives me this error: >>> import win32api Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: DLL load failed while importing win32api: The specified module could not be found. I'm on Windows 10 Home 64x. I've already tried pip install pypiwin32 And it successfully installs but nothing changes. I tried uninstalling and re-installing python as well. I also tried installing 'django' in the same way and it actually works when I import django, so I think it's a win32api issue only. >>> import win32api I expect the output to be none, but the actual output is always that error ^^

Output: Run Scripts\pywin32_postinstall.py -install in an Admin command prompt ref: https://github.com/mhammond/pywin32/issues/1431 edit: User @JoyfulPanda gave a warning: Running this script with admin rights will also copy pythoncom37.dll, pywintypes37.dll (corresponding to the pywin32 version), into C:\WINDOWS\system32, which effectively overwrites the corresponding DLL versions from Anaconda already there. This later causes problem when openning (on Windows) "Start Menu > Anaconda3 (64-bit) > Anaconda Prompt (a_virtual_env_name)". At least Anaconda 2019.07 has pywin32 223 installed by default. Pywin32 224 may work, but 225-228 causes problem for Anaconda (2019.07)


Instruction:
Open web in new tab Selenium + Python

So I am trying to open websites on new tabs inside my WebDriver. I want to do this, because opening a new WebDriver for each website takes about 3.5secs using PhantomJS, I want more speed... I'm using a multiprocess python script, and I want to get some elements from each page, so the workflow is like this: Open Browser Loop throught my array For element in array -> Open website in new tab -> do my business -> close it But I can't find any way to achieve this. Here's the code I'm using. It takes forever between websites, I need it to be fast... Other tools are allowed, but I don't know too many tools for scrapping website content that loads with JavaScript (divs created when some event is triggered on load etc) That's why I need Selenium... BeautifulSoup can't be used for some of my pages. #!/usr/bin/env python import multiprocessing, time, pika, json, traceback, logging, sys, os, itertools, urllib, urllib2, cStringIO, mysql.connector, shutil, hashlib, socket, urllib2, re from selenium import webdriver from selenium.webdriver.common.keys import Keys from PIL import Image from os import listdir from os.path import isfile, join from bs4 import BeautifulSoup from pprint import pprint def getPhantomData(parameters): try: # We create WebDriver browser = webdriver.Firefox() # Navigate to URL browser.get(parameters['target_url']) # Find all links by Selector links = browser.find_elements_by_css_selector(parameters['selector']) result = [] for link in links: # Extract link attribute and append to our list result.append(link.get_attribute(parameters['attribute'])) browser.close() browser.quit() return json.dumps({'data': result}) except Exception, err: browser.close() browser.quit() print err def callback(ch, method, properties, body): parameters = json.loads(body) message = getPhantomData(parameters) if message['data']: ch.basic_ack(delivery_tag=method.delivery_tag) else: ch.basic_reject(delivery_tag=method.delivery_tag, requeue=True) def consume(): credentials = pika.PlainCredentials('invitado', 'invitado') rabbit = pika.ConnectionParameters('localhost',5672,'/',credentials) connection = pika.BlockingConnection(rabbit) channel = connection.channel() # Conectamos al canal channel.queue_declare(queue='com.stuff.images', durable=True) channel.basic_consume(callback,queue='com.stuff.images') print ' [*] Waiting for messages. To exit press CTRL^C' try: channel.start_consuming() except KeyboardInterrupt: pass workers = 5 pool = multiprocessing.Pool(processes=workers) for i in xrange(0, workers): pool.apply_async(consume) try: while True: continue except KeyboardInterrupt: print ' [*] Exiting...' pool.terminate() pool.join()

Output: Editor's note: This answer no longer works for new Selenium versions. Refer to this comment. You can achieve the opening/closing of a tab by the combination of keys COMMAND + T or COMMAND + W (OSX). On other OSs you can use CONTROL + T / CONTROL + W. In selenium you can emulate such behavior. You will need to create one webdriver and as many tabs as the tests you need. Here it is the code. from selenium import webdriver from selenium.webdriver.common.keys import Keys driver = webdriver.Firefox() driver.get("http://www.google.com/") #open tab driver.find_element_by_tag_name('body').send_keys(Keys.COMMAND + 't') # You can use (Keys.CONTROL + 't') on other OSs # Load a page driver.get('http://stackoverflow.com/') # Make the tests... # close the tab # (Keys.CONTROL + 'w') on other OSs. driver.find_element_by_tag_name('body').send_keys(Keys.COMMAND + 'w') driver.close()


Instruction:
Change figure size and figure format in matplotlib

I want to obtain fig1 exactly of 4 by 3 inch sized, and in tiff format correcting the program below: import matplotlib.pyplot as plt list1 = [3,4,5,6,9,12] list2 = [8,12,14,15,17,20] plt.plot(list1, list2) plt.savefig('fig1.png', dpi = 300) plt.close()

Output: You can set the figure size if you explicitly create the figure with plt.figure(figsize=(3,4)) You need to set figure size before calling plt.plot() To change the format of the saved figure just change the extension in the file name. However, I don't know if any of matplotlib backends support tiff


Instruction:
python requests.get() returns improperly decoded text instead of UTF-8?

When the content-type of the server is 'Content-Type:text/html', requests.get() returns improperly encoded data. However, if we have the content type explicitly as 'Content-Type:text/html; charset=utf-8', it returns properly encoded data. Also, when we use urllib.urlopen(), it returns properly encoded data. Has anyone noticed this before? Why does requests.get() behave like this?

Output: From requests documentation: When you make a request, Requests makes educated guesses about the encoding of the response based on the HTTP headers. The text encoding guessed by Requests is used when you access r.text. You can find out what encoding Requests is using, and change it, using the r.encoding property. >>> r.encoding 'utf-8' >>> r.encoding = 'ISO-8859-1' Check the encoding requests used for your page, and if it's not the right one - try to force it to be the one you need. Regarding the differences between requests and urllib.urlopen - they probably use different ways to guess the encoding. Thats all.


Instruction:
How to get image width and height in OpenCV?

I want to get image width and height, how can I do that in OpenCV? For example: Mat src = imread("path_to_image"); cout << src.width; Is that right?

Output: You can use rows and cols: cout << "Width : " << src.cols << endl; cout << "Height: " << src.rows << endl; or size(): cout << "Width : " << src.size().width << endl; cout << "Height: " << src.size().height << endl; or size cout << "Width : " << src.size[1] << endl; cout << "Height: " << src.size[0] << endl;


Instruction:
Function to determine if two numbers are nearly equal when rounded to n significant decimal digits

I have been asked to test a library provided by a 3rd party. The library is known to be accurate to n significant figures. Any less-significant errors can safely be ignored. I want to write a function to help me compare the results: def nearlyequal(a, b, sigfig=5): The purpose of this function is to determine if two floating-point numbers (a and b) are approximately equal. The function will return True if a==b (exact match) or if a and b have the same value when rounded to sigfig significant-figures when written in decimal. Can anybody suggest a good implementation? I've written a mini unit-test. Unless you can see a bug in my tests then a good implementation should pass the following: assert nearlyequal(1, 1, 5) assert nearlyequal(1.0, 1.0, 5) assert nearlyequal(1.0, 1.0, 5) assert nearlyequal(-1e-9, 1e-9, 5) assert nearlyequal(1e9, 1e9 + 1 , 5) assert not nearlyequal(1e4, 1e4 + 1, 5) assert nearlyequal(0.0, 1e-15, 5) assert not nearlyequal(0.0, 1e-4, 6) Additional notes: Values a and b might be of type int, float or numpy.float64. Values a and b will always be of the same type. It's vital that conversion does not introduce additional error into the function. Lets keep this numerical, so functions that convert to strings or use non-mathematical tricks are not ideal. This program will be audited by somebody who is a mathematician who will want to be able to prove that the function does what it is supposed to do. Speed... I've got to compare a lot of numbers so the faster the better. I've got numpy, scipy and the standard-library. Anything else will be hard for me to get, especially for such a small part of the project.

Output: There is a function assert_approx_equal in numpy.testing (source here) which may be a good starting point. def assert_approx_equal(actual, desired, significant=7, err_msg='', verbose=True): """ Raise an assertion if two items are not equal up to significant digits. .. note:: It is recommended to use one of `assert_allclose`, `assert_array_almost_equal_nulp` or `assert_array_max_ulp` instead of this function for more consistent floating point comparisons. Given two numbers, check that they are approximately equal. Approximately equal is defined as the number of significant digits that agree.


Instruction:
Getting Google Spreadsheet CSV into A Pandas Dataframe

I uploaded a file to Google spreadsheets (to make a publically accessible example IPython Notebook, with data) I was using the file in it's native form could be read into a Pandas Dataframe. So now I use the following code to read the spreadsheet, works fine but just comes in as string,, and I'm not having any luck trying to get it back into a dataframe (you can get the data) import requests r = requests.get('https://docs.google.com/spreadsheet/ccc?key=0Ak1ecr7i0wotdGJmTURJRnZLYlV3M2daNTRubTdwTXc&output=csv') data = r.content The data ends up looking like: (1st row headers) ',City,region,Res_Comm,mkt_type,Quradate,National_exp,Alabama_exp,Sales_exp,Inventory_exp,Price_exp,Credit_exp\n0,Dothan,South_Central-Montgomery-Auburn-Wiregrass-Dothan,Residential,Rural,1/15/2010,2,2,3,2,3,3\n10,Foley,South_Mobile-Baldwin,Residential,Suburban_Urban,1/15/2010,4,4,4,4,4,3\n12,Birmingham,North_Central-Birmingham-Tuscaloosa-Anniston,Commercial,Suburban_Urban,1/15/2010,2,2,3,2,2,3\n The native pandas code that brings in the disk resident file looks like: df = pd.io.parsers.read_csv('/home/tom/Dropbox/Projects/annonallanswerswithmaster1012013.csv',index_col=0,parse_dates=['Quradate']) A "clean" solution would be helpful to many to provide an easy way to share datasets for Pandas use! I tried a bunch of alternative with no success and I'm pretty sure I'm missing something obvious again. Just a Update note The new Google spreadsheet has a different URL pattern Just use this in place of the URL in the above example and or the below answer and you should be fine here is an example: https://docs.google.com/spreadsheets/d/177_dFZ0i-duGxLiyg6tnwNDKruAYE-_Dd8vAQziipJQ/export?format=csv&id see solution below from @Max Ghenis which just used pd.read_csv, no need for StringIO or requests...

Output: You can use read_csv() on a StringIO object: from io import BytesIO import requests import pandas as pd r = requests.get('https://docs.google.com/spreadsheet/ccc?key=0Ak1ecr7i0wotdGJmTURJRnZLYlV3M2daNTRubTdwTXc&output=csv') data = r.content In [10]: df = pd.read_csv(BytesIO(data), index_col=0,parse_dates=['Quradate']) In [11]: df.head() Out[11]: City region Res_Comm \ 0 Dothan South_Central-Montgomery-Auburn-Wiregrass-Dothan Residential 10 Foley South_Mobile-Baldwin Residential 12 Birmingham North_Central-Birmingham-Tuscaloosa-Anniston Commercial 38 Brent North_Central-Birmingham-Tuscaloosa-Anniston Residential 44 Athens North_Huntsville-Decatur-Florence Residential mkt_type Quradate National_exp Alabama_exp Sales_exp \ 0 Rural 2010-01-15 00:00:00 2 2 3 10 Suburban_Urban 2010-01-15 00:00:00 4 4 4 12 Suburban_Urban 2010-01-15 00:00:00 2 2 3 38 Rural 2010-01-15 00:00:00 3 3 3 44 Suburban_Urban 2010-01-15 00:00:00 4 5 4 Inventory_exp Price_exp Credit_exp 0 2 3 3 10 4 4 3 12 2 2 3 38 3 3 2 44 4 4 4


Instruction:
Remove substring only at the end of string

I have a bunch of strings, some of them have ' rec'. I want to remove that only if those are the last 4 characters. So in other words I have somestring = 'this is some string rec' and I want it to become somestring = 'this is some string' What is the Python way to approach this?

Output: def rchop(s, suffix): if suffix and s.endswith(suffix): return s[:-len(suffix)] return s somestring = 'this is some string rec' rchop(somestring, ' rec') # returns 'this is some string'


Instruction:
Tree plotting in Python

I want to plot trees using Python. Decision trees, Organizational charts, etc. Any library that helps me with that?

Output: There's graphviz - http://www.graphviz.org/. It uses the "DOT" language to plot graphs. You can either generate the DOT code yourself, or use pydot - https://github.com/pydot/pydot. You could also use networkx - http://networkx.lanl.gov/tutorial/tutorial.html#drawing-graphs, which make it easy to draw to either graphviz or matplotlib. networkx + matplotlib + graphviz gives you the most flexibility and power, but you need to install a lot. If you want a quick solution, try: Install Graphviz. open('hello.dot','w').write("digraph G {Hello->World}") import subprocess subprocess.call(["path/to/dot.exe","-Tpng","hello.dot","-o","graph1.png"]) # I think this is right - try it form the command line to debug Then you install pydot, because pydot already does this for you. Then you can use networkx to "drive" pydot.


Instruction:
Write unittest for console print

Function foo prints to console. I want to test the console print. How can I achieve this in Python? I need to test this function, which has no return statement: def foo(inStr): print "hi"+inStr My test: def test_foo(): cmdProcess = subprocess.Popen(foo("test"), stdout=subprocess.PIPE) cmdOut = cmdProcess.communicate()[0] self.assertEquals("hitest", cmdOut)

Output: You can easily capture standard output by just temporarily redirecting sys.stdout to a StringIO object, as follows: import StringIO import sys def foo(inStr): print "hi"+inStr def test_foo(): capturedOutput = StringIO.StringIO() # Make StringIO. sys.stdout = capturedOutput # Redirect stdout. foo('test') # Call function. sys.stdout = sys.__stdout__ # Reset redirect. print 'Captured', capturedOutput.getvalue() # Now works. test_foo() The output of this program is: Captured hitest This shows that the redirection successfully captured the output and that you were able to restore the output stream to what it was before you began the capture. Note that the code above in for Python 2.7, as the question indicates. Python 3 is slightly different: import io import sys def foo(inStr): print ("hi"+inStr) def test_foo(): capturedOutput = io.StringIO() # Create StringIO. sys.stdout = capturedOutput # Redirect stdout. foo('test') # Call function. sys.stdout = sys.__stdout__ # Reset redirect. print ('Captured', capturedOutput.getvalue()) # Now works. test_foo() You also need to be careful if your function can raise an exception since that may mean that the redirection reset may not be done. This may then affect later tests. One way to solve that would be to wrap the function call in a try/except block and, on exception, reset the redirection and re-throw the exception. The single-line call to your function would then become something like this, with the other code remaining as-is: try: foo('test') # Call function. except: sys.stdout = sys.__stdout__ # Reset redirect. raise


Instruction:
Converting an OpenCV Image to Black and White

How do you convert a grayscale OpenCV image to black and white? I see a similar question has already been asked, but I'm using OpenCV 2.3, and the proposed solution no longer seems to work. I'm trying to convert a greyscale image to black and white, so that anything not absolutely black is white, and use this as a mask for surf.detect(), in order to ignore keypoints found on the edge of the black mask area. The following Python gets me almost there, but the threshold value sent to Threshold() doesn't appear to have any effect. If I set it to 0 or 16 or 128 or 255, the result is the same, with all pixels with a value > 128 becoming white, and everything else becoming black. What am I doing wrong? import cv, cv2 fn = 'myfile.jpg' im_gray = cv2.imread(fn, cv.CV_LOAD_IMAGE_GRAYSCALE) im_gray_mat = cv.fromarray(im_gray) im_bw = cv.CreateImage(cv.GetSize(im_gray_mat), cv.IPL_DEPTH_8U, 1); im_bw_mat = cv.GetMat(im_bw) threshold = 0 # 128#255# HAS NO EFFECT!?!? cv.Threshold(im_gray_mat, im_bw_mat, threshold, 255, cv.CV_THRESH_BINARY | cv.CV_THRESH_OTSU); cv2.imshow('', np.asarray(im_bw_mat)) cv2.waitKey()

Output: Step-by-step answer similar to the one you refer to, using the new cv2 Python bindings: 1. Read a grayscale image import cv2 im_gray = cv2.imread('grayscale_image.png', cv2.IMREAD_GRAYSCALE) 2. Convert grayscale image to binary (thresh, im_bw) = cv2.threshold(im_gray, 128, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU) which determines the threshold automatically from the image using Otsu's method, or if you already know the threshold you can use: thresh = 127 im_bw = cv2.threshold(im_gray, thresh, 255, cv2.THRESH_BINARY)[1] 3. Save to disk cv2.imwrite('bw_image.png', im_bw)


Instruction:
RemoveError: 'requests' is a dependency of conda and cannot be removed from conda's operating environment

Having trouble installing a third party library and I have not seen this error before using Windows 10 with Anaconda installed: C:\Users\XYZ>conda env create -f python3.6-environment-windows.yml Collecting package metadata: done Solving environment: done Downloading and Extracting Packages certifi-2018.1.18 | 144 KB | ############################################################################ | 100% mkl-2018.0.1 | 155.2 MB | ############################################################################ | 100% pytz-2018.9 | 229 KB | ############################################################################ | 100% icc_rt-2019.0.0 | 9.4 MB | ############################################################################ | 100% icu-58.2 | 21.8 MB | ############################################################################ | 100% pip-9.0.1 | 1.7 MB | ############################################################################ | 100% xz-5.2.3 | 348 KB | ############################################################################ | 100% sip-4.18.1 | 269 KB | ############################################################################ | 100% libpng-1.6.36 | 1.3 MB | ############################################################################ | 100% vc-14 | 985 B | ############################################################################ | 100% numpy-1.14.0 | 3.7 MB | ############################################################################ | 100% python-3.6.4 | 17.6 MB | ############################################################################ | 100% jpeg-9c | 314 KB | ############################################################################ | 100% wheel-0.30.0 | 85 KB | ############################################################################ | 100% wincertstore-0.2 | 13 KB | ############################################################################ | 100% freetype-2.9.1 | 475 KB | ############################################################################ | 100% scipy-1.0.0 | 13.0 MB | ############################################################################ | 100% pyparsing-2.3.1 | 54 KB | ############################################################################ | 100% kiwisolver-1.0.1 | 60 KB | ############################################################################ | 100% qt-5.6.2 | 55.6 MB | ############################################################################ | 100% python-dateutil-2.7. | 218 KB | ############################################################################ | 100% vs2015_runtime-14.0. | 1.9 MB | ############################################################################ | 100% ca-certificates-2017 | 489 KB | ############################################################################ | 100% tk-8.6.7 | 3.5 MB | ############################################################################ | 100% setuptools-38.4.0 | 540 KB | ############################################################################ | 100% matplotlib-2.2.2 | 6.5 MB | ############################################################################ | 100% six-1.12.0 | 21 KB | ############################################################################ | 100% openssl-1.0.2n | 5.4 MB | ############################################################################ | 100% pyqt-5.6.0 | 4.5 MB | ############################################################################ | 100% zlib-1.2.11 | 236 KB | ############################################################################ | 100% tornado-5.1.1 | 665 KB | ############################################################################ | 100% sqlite-3.22.0 | 907 KB | ############################################################################ | 100% cycler-0.10.0 | 8 KB | ############################################################################ | 100% Preparing transaction: done Verifying transaction: failed RemoveError: 'requests' is a dependency of conda and cannot be removed from conda's operating environment. RemoveError: 'setuptools' is a dependency of conda and cannot be removed from conda's operating environment. In reference to the instructions here - https://enigma.co/catalyst/install.html#installing-with-conda

Output: running conda update conda before solved the problem for me


Instruction:
How can I annotate a generator in Python 3?

Python 3.x supports (optional) function annotations: def add_ints(x: int, y: int) -> int: return x + y I sometimes encounter problems as to how to represent a given "type", and this time, I have a function that returns a generator: def myfunc(x: [int]) -> "generator that returns ints": # ^~~~~~~~~~~~~~~~~~~~~~~~~~ return (n for n in x if n % 2 == 0) How should I annotate the return value?

Output: The collections.abc module defines the collections.abc.Generator type, which you can use like: Generator[yield_type, send_type, return_type] where send_type and return_type default to None. (If your generator neither sends nor returns, it can just be annotated as Generator[yield_type]. Though it has been deprecated since 3.9, typing.Generator behaves very similarly.


Instruction:
List Directories and get the name of the Directory

I am trying to get the code to list all the directories in a folder, change directory into that folder and get the name of the current folder. The code I have so far is below and isn't working at the minute. I seem to be getting the parent folder name. import os for directories in os.listdir(os.getcwd()): dir = os.path.join('/home/user/workspace', directories) os.chdir(dir) current = os.path.dirname(dir) new = str(current).split("-")[0] print new I also have other files in the folder but I do not want to list them. I have tried the below code but I haven't got it working yet either. for directories in os.path.isdir(os.listdir(os.getcwd())): Can anyone see where I am going wrong? Thanks Got it working but it seems a bit round about. import os os.chdir('/home/user/workspace') all_subdirs = [d for d in os.listdir('.') if os.path.isdir(d)] for dirs in all_subdirs: dir = os.path.join('/home/user/workspace', dirs) os.chdir(dir) current = os.getcwd() new = str(current).split("/")[4] print new

Output: This will print all the subdirectories of the current directory: print ([name for name in os.listdir(".") if os.path.isdir(name)]) I'm not sure what you're doing with split("-"), but perhaps this code will help you find a solution? If you want the full pathnames of the directories, use abspath: print ([os.path.abspath(name) for name in os.listdir(".") if os.path.isdir(name)]) # needs outside parenthesis Note that these pieces of code will only get the immediate subdirectories. If you want sub-sub-directories and so on, you should use walk as others have suggested.


Instruction:
getting sheet names from openpyxl

I have a moderately large xlsx file (around 14 MB) and OpenOffice hangs trying to open it. I was trying to use openpyxl to read the content, following this tutorial. The code snippet is as follows: from openpyxl import load_workbook wb = load_workbook(filename = 'large_file.xlsx', use_iterators = True) ws = wb.get_sheet_by_name(name = 'big_data') The problem is, I don't know the sheet name, and Sheet1/Sheet2.. etc. didn't work (returned NoneType object). I could not find a documentation telling me How to get the sheet names for an xlsx files using openpyxl. Can anyone help me?

Output: Use the sheetnames property: sheetnames Returns the list of the names of worksheets in this workbook. Names are returned in the worksheets order. Type: list of strings print (wb.sheetnames) You can also get worksheet objects from wb.worksheets: ws = wb.worksheets[0]


Instruction:
Getting console.log output from Chrome with Selenium Python API bindings

I'm using Selenium to run tests in Chrome via the Python API bindings, and I'm having trouble figuring out how to configure Chrome to make the console.log output from the loaded test available. I see that there are get_log() and log_types() methods on the WebDriver object, and I've seen Get chrome's console log which shows how to do things in Java. But I don't see an equivalent of Java's LoggingPreferences type in the Python API. Is there some way to accomplish what I need?

Output: Ok, finally figured it out: from selenium import webdriver from selenium.webdriver.common.desired_capabilities import DesiredCapabilities # enable browser logging d = DesiredCapabilities.CHROME d['loggingPrefs'] = { 'browser':'ALL' } driver = webdriver.Chrome(desired_capabilities=d) # load the desired webpage driver.get('http://foo.com') # print messages for entry in driver.get_log('browser'): print(entry) Entries whose source field equals 'console-api' correspond to console messages, and the message itself is stored in the message field. Starting from chromedriver, 75.0.3770.8, you have to use goog:loggingPrefs instead of loggingPrefs: d['goog:loggingPrefs'] = { 'browser':'ALL' }


Instruction:
What does = (equal) do in f-strings inside the expression curly brackets?

The usage of {} in Python f-strings is well known to execute pieces of code and give the result in string format. However, what does the '=' at the end of the expression mean? log_file = open("log_aug_19.txt", "w") console_error = '...stuff...' # the real code generates it with regex log_file.write(f'{console_error=}')

Output: This is actually a brand-new feature as of Python 3.8. Added an = specifier to f-strings. An f-string such as f'{expr=}' will expand to the text of the expression, an equal sign, then the representation of the evaluated expression. Essentially, it facilitates the frequent use-case of print-debugging, so, whereas we would normally have to write: f"some_var={some_var}" we can now write: f"{some_var=}" So, as a demonstration, using a shiny-new Python 3.8.0 REPL: >>> print(f"{foo=}") foo=42 >>>


Instruction:
What is difference between pop() vs del for deleting a dictionary item? (python)

In Python there are at least two methods to delete an item from a dict using a key. d = {"keyA": 123, "keyB": 456, "keyC": 789} #remove via pop d.pop("keyA") #remove via del del d["keyB"] Both methods would remove the item from the dict. I wonder what the difference between these methods is and in what kinds of situations I should use one or the other.

Output: d.pop(key) -- consider using this if you need the value for the item being deleted, and/or you want to specify a default value and don't want an exception raised if the key doesn't exist. e.g., value = d.pop(key, None) del key -- consider using this if you are certain that the key exists (or you expect an exception if it doesn't), and you don't need the value of the item being deleted. e.g., del key From the official Python language documentation: del d[key] Remove d[key] from d. Raises a KeyError if key is not in the map. pop(key[, default]) If key is in the dictionary, remove it and return its value, else return default. If default is not given and key is not in the dictionary, a KeyError is raised. https://docs.python.org/3/library/stdtypes.html#mapping-types-dict


Instruction:
Pytest fixture for a class through self not as method argument

Often I'll write a test class that uses a pytest fixture in every method. Here's an example. I'd like to be able to avoid having to write the fixture name in the signature of every method. It's not DRY. How can this be done? I would like to be able to access the fixture by giving the fixture as an attribute of the test class. In this example, I would like to see the google fixture as an attribute of TestGoogle. Is this possible? from bs4 import BeautifulSoup import pytest import requests @pytest.fixture() def google(): return requests.get("https://www.google.com") class TestGoogle: def test_alive(self, google): assert google.status_code == 200 def test_html_title(self, google): soup = BeautifulSoup(google.content, "html.parser") assert soup.title.text.upper() == "GOOGLE"

Output: Sure, just use an autouse fixture. Here is the relevant spot in pytest docs. In your example, the change would be introducing an extra fixture (I named it _request_google_page): from bs4 import BeautifulSoup import pytest import requests @pytest.fixture() def google(): return requests.get("https://www.google.com") class TestGoogle: @pytest.fixture(autouse=True) def _request_google_page(self, google): self._response = google def test_alive(self): assert self._response.status_code == 200 def test_html_title(self): soup = BeautifulSoup(self._response.content, "html.parser") assert soup.title.text.upper() == "GOOGLE" You could even drop the google fixture completely and move the code to _request_google_page: @pytest.fixture(autouse=True) def _request_google_page(self): self._response = requests.get("https://www.google.com") Note that _request_google_page will be called once per test by default, so each test will get a new response. If you want the response to be initialized once and reused throughout all tests in the TestGoogle class, adjust the fixture scopes (scope='class' for _request_google_page and scope='module' or scope='session' for google). Example: from bs4 import BeautifulSoup import pytest import requests @pytest.fixture(scope='module') def google(): return requests.get("https://www.google.com") @pytest.fixture(autouse=True, scope='class') def _request_google_page(request, google): request.cls._response = google class TestGoogle: def test_alive(self): assert self._response.status_code == 200 def test_html_title(self): soup = BeautifulSoup(self._response.content, "html.parser") assert soup.title.text.upper() == "GOOGLE"


Instruction:
Fast punctuation removal with pandas

This is a self-answered post. Below I outline a common problem in the NLP domain and propose a few performant methods to solve it. Oftentimes the need arises to remove punctuation during text cleaning and pre-processing. Punctuation is defined as any character in string.punctuation: >>> import string string.punctuation '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~' This is a common enough problem and has been asked before ad nauseam. The most idiomatic solution uses pandas str.replace. However, for situations which involve a lot of text, a more performant solution may need to be considered. What are some good, performant alternatives to str.replace when dealing with hundreds of thousands of records?

Output: Setup For the purpose of demonstration, let's consider this DataFrame. df = pd.DataFrame({'text':['a..b?!??', '%hgh&12','abc123!!!', '$$$1234']}) df text 0 a..b?!?? 1 %hgh&12 2 abc123!!! 3 $$$1234 Below, I list the alternatives, one by one, in increasing order of performance str.replace This option is included to establish the default method as a benchmark for comparing other, more performant solutions. This uses pandas in-built str.replace function which performs regex-based replacement. df['text'] = df['text'].str.replace(r'[^\w\s]+', '') df text 0 ab 1 hgh12 2 abc123 3 1234 This is very easy to code, and is quite readable, but slow. regex.sub This involves using the sub function from the re library. Pre-compile a regex pattern for performance, and call regex.sub inside a list comprehension. Convert df['text'] to a list beforehand if you can spare some memory, you'll get a nice little performance boost out of this. import re p = re.compile(r'[^\w\s]+') df['text'] = [p.sub('', x) for x in df['text'].tolist()] df text 0 ab 1 hgh12 2 abc123 3 1234 Note: If your data has NaN values, this (as well as the next method below) will not work as is. See the section on "Other Considerations". str.translate python's str.translate function is implemented in C, and is therefore very fast. How this works is: First, join all your strings together to form one huge string using a single (or more) character separator that you choose. You must use a character/substring that you can guarantee will not belong inside your data. Perform str.translate on the large string, removing punctuation (the separator from step 1 excluded). Split the string on the separator that was used to join in step 1. The resultant list must have the same length as your initial column. Here, in this example, we consider the pipe separator |. If your data contains the pipe, then you must choose another separator. import string punct = '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{}~' # `|` is not present here transtab = str.maketrans(dict.fromkeys(punct, '')) df['text'] = '|'.join(df['text'].tolist()).translate(transtab).split('|') df text 0 ab 1 hgh12 2 abc123 3 1234 Performance str.translate performs the best, by far. Note that the graph below includes another variant Series.str.translate from MaxU's answer. (Interestingly, I reran this a second time, and the results are slightly different from before. During the second run, it seems re.sub was winning out over str.translate for really small amounts of data.) There is an inherent risk involved with using translate (particularly, the problem of automating the process of deciding which separator to use is non-trivial), but the trade-offs are worth the risk. Other Considerations Handling NaNs with list comprehension methods; Note that this method (and the next) will only work as long as your data does not have NaNs. When handling NaNs, you will have to determine the indices of non-null values and replace those only. Try something like this: df = pd.DataFrame({'text': [ 'a..b?!??', np.nan, '%hgh&12','abc123!!!', '$$$1234', np.nan]}) idx = np.flatnonzero(df['text'].notna()) col_idx = df.columns.get_loc('text') df.iloc[idx,col_idx] = [ p.sub('', x) for x in df.iloc[idx,col_idx].tolist()] df text 0 ab 1 NaN 2 hgh12 3 abc123 4 1234 5 NaN Dealing with DataFrames; If you are dealing with DataFrames, where every column requires replacement, the procedure is simple: v = pd.Series(df.values.ravel()) df[:] = translate(v).values.reshape(df.shape) Or, v = df.stack() v[:] = translate(v) df = v.unstack() Note that the translate function is defined below in with the benchmarking code. Every solution has tradeoffs, so deciding what solution best fits your needs will depend on what you're willing to sacrifice. Two very common considerations are performance (which we've already seen), and memory usage. str.translate is a memory-hungry solution, so use with caution. Another consideration is the complexity of your regex. Sometimes, you may want to remove anything that is not alphanumeric or whitespace. Othertimes, you will need to retain certain characters, such as hyphens, colons, and sentence terminators [.!?]. Specifying these explicitly add complexity to your regex, which may in turn impact the performance of these solutions. Make sure you test these solutions on your data before deciding what to use. Lastly, unicode characters will be removed with this solution. You may want to tweak your regex (if using a regex-based solution), or just go with str.translate otherwise. For even more performance (for larger N), take a look at this answer by Paul Panzer. Appendix Functions def pd_replace(df): return df.assign(text=df['text'].str.replace(r'[^\w\s]+', '')) def re_sub(df): p = re.compile(r'[^\w\s]+') return df.assign(text=[p.sub('', x) for x in df['text'].tolist()]) def translate(df): punct = string.punctuation.replace('|', '') transtab = str.maketrans(dict.fromkeys(punct, '')) return df.assign( text='|'.join(df['text'].tolist()).translate(transtab).split('|') ) # MaxU's version (https://stackoverflow.com/a/50444659/4909087) def pd_translate(df): punct = string.punctuation.replace('|', '') transtab = str.maketrans(dict.fromkeys(punct, '')) return df.assign(text=df['text'].str.translate(transtab)) Performance Benchmarking Code from timeit import timeit import pandas as pd import matplotlib.pyplot as plt res = pd.DataFrame( index=['pd_replace', 're_sub', 'translate', 'pd_translate'], columns=[10, 50, 100, 500, 1000, 5000, 10000, 50000], dtype=float ) for f in res.index: for c in res.columns: l = ['a..b?!??', '%hgh&12','abc123!!!', '$$$1234'] * c df = pd.DataFrame({'text' : l}) stmt = '{}(df)'.format(f) setp = 'from __main__ import df, {}'.format(f) res.at[f, c] = timeit(stmt, setp, number=30) ax = res.div(res.min()).T.plot(loglog=True) ax.set_xlabel("N"); ax.set_ylabel("time (relative)"); plt.show()


Instruction:
Thread vs. Threading

What's the difference between the threading and thread modules in Python?

Output: In Python 3, thread has been renamed to _thread. It is infrastructure code that is used to implement threading, and normal Python code shouldn't be going anywhere near it. _thread exposes a fairly raw view of the underlying OS level processes. This is almost never what you want, hence the rename in Py3k to indicate that it is really just an implementation detail. threading adds some additional automatic accounting, as well as several convenience utilities, all of which makes it the preferred option for standard Python code. Note: as mentioned in Jeril's alternate answer, when considering the use of threads in Python for execution of background tasks, also take a look at concurrent.futures.ThreadPoolExecutor as an even higher level API than threading: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor


Instruction:
How to give a Pydantic list field a default value?

I want to create a Pydantic model in which there is a list field, which left uninitialized has a default value of an empty list. Is there an idiomatic way to do this? For Python's built-in dataclass objects you can use field(default_factory=list), however in my own experiments this seems to prevent my Pydantic models from being pickled. A naive implementation might be, something like this: from pydantic import BaseModel class Foo(BaseModel): defaulted_list_field: Sequence[str] = [] # Bad! But we all know not to use a mutable value like the empty-list literal as a default. So what's the correct way to give a Pydantic list-field a default value?

Output: For pydantic you can use mutable default value, like: class Foo(BaseModel): defaulted_list_field: List[str] = [] f1, f2 = Foo(), Foo() f1.defaulted_list_field.append("hey!") print(f1) # defaulted_list_field=['hey!'] print(f2) # defaulted_list_field=[] It will be handled correctly (deep copy) and each model instance will have its own empty list. See "Fields with non-hashable default values" from the documentation. Pydantic also has default_factory parameter. In the case of an empty list, the result will be identical, it is rather used when declaring a field with a default value, you may want it to be dynamic (i.e. different for each model). from typing import List from pydantic import BaseModel, Field from uuid import UUID, uuid4 class Foo(BaseModel): defaulted_list_field: List[str] = Field(default_factory=list) uid: UUID = Field(default_factory=uuid4)


Instruction:
Show default value for editing on Python input possible?

Is it possible for python to accept input like this: Folder name: Download But instead of the user typing "Download" it is already there as a initial value. If the user wants to edit it as "Downloads" all he has to do is add a 's' and press enter. Using normal input command: folder=input('Folder name: ') all I can get is a blank prompt: Folder name: Is there a simple way to do this that I'm missing?

Output: The standard library functions input() and raw_input() don't have this functionality. If you're using Linux you can use the readline module to define an input function that uses a prefill value and advanced line editing: import readline def rlinput(prompt, prefill=''): readline.set_startup_hook(lambda: readline.insert_text(prefill)) try: return input(prompt) # or raw_input in Python 2 finally: readline.set_startup_hook()


Instruction:
Pandas replace a character in all column names

I have data frames with column names (coming from .csv files) containing ( and ) and I'd like to replace them with _. How can I do that in place for all columns?

Output: Use str.replace: df.columns = df.columns.str.replace("[()]", "_", regex=True) Sample: df = pd.DataFrame({'(A)':[1,2,3], '(B)':[4,5,6], 'C)':[7,8,9]}) print (df) (A) (B) C) 0 1 4 7 1 2 5 8 2 3 6 9 df.columns = df.columns.str.replace(r"[()]", "_", regex=True) print (df) _A_ _B_ C_ 0 1 4 7 1 2 5 8 2 3 6 9