Instruction:
Convert Pandas dataframe to csv string

Here is an example of what I am trying to get: I have: import pandas as pd df = pd.DataFrame({'A' : [0, 1], 'B' : [1, 6]}) My goal is: ',A,B\n0,0,1\n1,1,6\n' I can achieve this with lazy and horrible: df.to_csv('temp.csv') # create unnecessary file body = open('temp.csv').read() Also to_string() methods looks very promising; however, the best I can come up with is this: body = df.to_string()[1:].replace(' ', ',') + '\n' This does not create an unnecessary file, but seems sloppy and perhaps not very reliable. Am I missing a simpler solution?

Output: In [10]: df = pd.DataFrame({'A' : [0, 1], 'B' : [1, 6]}) In [11]: import io In [12]: s = io.StringIO() In [13]: df.to_csv(s) In [14]: s.getvalue() Out[14]: ',A,B\n0,0,1\n1,1,6\n'


Instruction:
What do these operators mean (** , ^ , %, //)?

Other than the standard +, -, *and / operators; but what does these mean (** , ^ , %, //) ? >>> 9+float(2) # addition 11.0 >>> 9-float(2) # subtraction 7.0 >>> 9*float(2) # multiplication 18.0 >>> 9/float(2) # division 4.5 >>> >>> 9**float(2) # This looks like a square, (i.e. power 2) 81.0 >>> 9**float(3) # So ** is equivalent to `math.pow(x,p)` ? 729.0 How about the ^ operator? >>> 9^int(2) # What is `^` in `x^u` , it only allows `int` for `u` 11 >>> 9^int(3) 10 >>> 9^int(4) 13 >>> 9^int(5) 12 >>> 9^int(6) 15 >>> 9^int(7) 14 >>> 9^int(8) 1 >>> 9^int(9) 0 >>> 9^int(10) 3 >>> 9^int(11) 2 >>> 9^int(12) 5 % in x%m returns a normal remainder modulus, but only if m < x, why is that so? What does % do? >>> 9%float(2) 1.0 >>> 9%float(3) 0.0 >>> 9%float(4) 1.0 >>> 9%float(5) 4.0 >>> 9%float(6) 3.0 >>> 9%float(7) 2.0 >>> 9%float(8) 1.0 >>> 9%float(9) 0.0 >>> 9%float(10) 9.0 >>> 9%float(11) 9.0 >>> 9%float(12) 9.0 How about the // operator? what does it do? >>> 9//float(2) 4.0 >>> 9//float(3) 3.0 >>> 9//float(4) 2.0 >>> 9//float(5) 1.0 >>> 9//float(6) 1.0 >>> 9//float(7) 1.0 >>> 9//float(8) 1.0 >>> 9//float(9) 1.0 >>> 9//float(1) 9.0 >>> 9//float(0.5) 18.0

Output: **: exponentiation ^: exclusive-or (bitwise) %: modulus //: divide with integral result (discard remainder)


Instruction:
how to remove an element in lxml

I need to completely remove elements, based on the contents of an attribute, using python's lxml. Example: import lxml.etree as et xml=""" <groceries> <fruit state="rotten">apple</fruit> <fruit state="fresh">pear</fruit> <fruit state="fresh">starfruit</fruit> <fruit state="rotten">mango</fruit> <fruit state="fresh">peach</fruit> </groceries> """ tree=et.fromstring(xml) for bad in tree.xpath("//fruit[@state=\'rotten\']"): #remove this element from the tree print et.tostring(tree, pretty_print=True) I would like this to print: <groceries> <fruit state="fresh">pear</fruit> <fruit state="fresh">starfruit</fruit> <fruit state="fresh">peach</fruit> </groceries> Is there a way to do this without storing a temporary variable and printing to it manually, as: newxml="<groceries>\n" for elt in tree.xpath('//fruit[@state=\'fresh\']'): newxml+=et.tostring(elt) newxml+="</groceries>"

Output: Use the remove method of an xmlElement : tree=et.fromstring(xml) for bad in tree.xpath("//fruit[@state=\'rotten\']"): bad.getparent().remove(bad) # here I grab the parent of the element to call the remove directly on it print et.tostring(tree, pretty_print=True, xml_declaration=True) If I had to compare with the @Acorn version, mine will work even if the elements to remove are not directly under the root node of your xml.


Instruction:
pip says: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes

I am using pip to install all my python packages but get error as shown in the trace below. What is the problem and how can I solve it? usr@comp:~$ pip install flask Collecting flask Using cached Flask-0.11.1-py2.py3-none-any.whl Collecting itsdangerous>=0.21 (from flask) Using cached itsdangerous-0.24.tar.gz Collecting click>=2.0 (from flask) Using cached click-6.6.tar.gz Collecting Werkzeug>=0.7 (from flask) Using cached Werkzeug-0.11.11-py2.py3-none-any.whl Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.4 in /usr/lib/python2.7/dist-packages (from flask) Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in /usr/lib/python2.7/dist-packages (from Jinja2>=2.4->flask) THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. Werkzeug>=0.7 from https://pypi.python.org/packages/a9/5e/41f791a3f380ec50f2c4c3ef1399d9ffce6b4fe9a7f305222f014cf4fe83/Werkzeug-0.11.11-py2.py3-none-any.whl#md5=c63a21eedce9504d223ed89358c4bdc9 (from flask): Expected md5 c63a21eedce9504d223ed89358c4bdc9 Got 13a168aafcc43354b6c79ef44bb0dc71

Output: There is a similar problem (Why does pip fail with bad md5 hash for package?) from 2013 the solution that I tried that worked for me is this: sudo pip install --no-cache-dir flask given by attolee


Instruction:
installing python packages without internet and using source code as .tar.gz and .whl

we are trying to install couple of python packages without internet. For ex : python-keystoneclient For that we have the packages downloaded from https://pypi.python.org/pypi/python-keystoneclient/1.7.1 and kept it in server. However, while installing tar.gz and .whl packages , the installation is looking for dependent packages to be installed first. Since there is no internet connection in the server, it is getting failed. For ex : For python-keystoneclient we have the following dependent packages stevedore (>=1.5.0) six (>=1.9.0) requests (>=2.5.2) PrettyTable (<0.8,>=0.7) oslo.utils (>=2.0.0) oslo.serialization (>=1.4.0) oslo.i18n (>=1.5.0) oslo.config (>=2.3.0) netaddr (!=0.7.16,>=0.7.12) debtcollector (>=0.3.0) iso8601 (>=0.1.9) Babel (>=1.3) argparse pbr (<2.0,>=1.6) When i try to install packages one by one from the above list, once again its looking for nested dependency . Is there any way we could list ALL the dependent packages for installing a python module like python-keystoneclient.

Output: This is how I handle this case: On the machine where I have access to Internet: mkdir keystone-deps pip download python-keystoneclient -d "/home/aviuser/keystone-deps" tar cvfz keystone-deps.tgz keystone-deps Then move the tar file to the destination machine that does not have Internet access and perform the following: tar xvfz keystone-deps.tgz cd keystone-deps pip install python_keystoneclient-2.3.1-py2.py3-none-any.whl -f ./ --no-index You may need to add --no-deps to the command as follows: pip install python_keystoneclient-2.3.1-py2.py3-none-any.whl -f ./ --no-index --no-deps


Instruction:
How to stop a looping thread in Python?

What's the proper way to tell a looping thread to stop looping? I have a fairly simple program that pings a specified host in a separate threading.Thread class. In this class it sleeps 60 seconds, the runs again until the application quits. I'd like to implement a 'Stop' button in my wx.Frame to ask the looping thread to stop. It doesn't need to end the thread right away, it can just stop looping once it wakes up. Here is my threading class (note: I haven't implemented looping yet, but it would likely fall under the run method in PingAssets) class PingAssets(threading.Thread): def __init__(self, threadNum, asset, window): threading.Thread.__init__(self) self.threadNum = threadNum self.window = window self.asset = asset def run(self): config = controller.getConfig() fmt = config['timefmt'] start_time = datetime.now().strftime(fmt) try: if onlinecheck.check_status(self.asset): status = "online" else: status = "offline" except socket.gaierror: status = "an invalid asset tag." msg =("{}: {} is {}. \n".format(start_time, self.asset, status)) wx.CallAfter(self.window.Logger, msg) And in my wxPyhton Frame I have this function called from a Start button: def CheckAsset(self, asset): self.count += 1 thread = PingAssets(self.count, asset, self) self.threads.append(thread) thread.start()

Output: Threaded stoppable function Instead of subclassing threading.Thread, one can modify the function to allow stopping by a flag. We need an object, accessible to running function, to which we set the flag to stop running. We can use threading.currentThread() object. import threading import time def doit(arg): t = threading.currentThread() while getattr(t, "do_run", True): print ("working on %s" % arg) time.sleep(1) print("Stopping as you wish.") def main(): t = threading.Thread(target=doit, args=("task",)) t.start() time.sleep(5) t.do_run = False if __name__ == "__main__": main() The trick is, that the running thread can have attached additional properties. The solution builds on assumptions: the thread has a property "do_run" with default value True driving parent process can assign to started thread the property "do_run" to False. Running the code, we get following output: $ python stopthread.py working on task working on task working on task working on task working on task Stopping as you wish. Pill to kill - using Event Other alternative is to use threading.Event as function argument. It is by default False, but external process can "set it" (to True) and function can learn about it using wait(timeout) function. We can wait with zero timeout, but we can also use it as the sleeping timer (used below). def doit(stop_event, arg): while not stop_event.wait(1): print ("working on %s" % arg) print("Stopping as you wish.") def main(): pill2kill = threading.Event() t = threading.Thread(target=doit, args=(pill2kill, "task")) t.start() time.sleep(5) pill2kill.set() t.join() Edit: I tried this in Python 3.6. stop_event.wait() blocks the event (and so the while loop) until release. It does not return a boolean value. Using stop_event.is_set() works instead. Stopping multiple threads with one pill Advantage of pill to kill is better seen, if we have to stop multiple threads at once, as one pill will work for all. The doit will not change at all, only the main handles the threads a bit differently. def main(): pill2kill = threading.Event() tasks = ["task ONE", "task TWO", "task THREE"] def thread_gen(pill2kill, tasks): for task in tasks: t = threading.Thread(target=doit, args=(pill2kill, task)) yield t threads = list(thread_gen(pill2kill, tasks)) for thread in threads: thread.start() time.sleep(5) pill2kill.set() for thread in threads: thread.join()


Instruction:
How to pass a list from Python, by Jinja2 to JavaScript

Let's say I have a Python variable: list_of_items = ['1','2','3','4','5'] and I pass it to Jinja by rendering HTML, and I also have a function in JavaScript called somefunction(variable). I am trying to pass each item of list_of_items. I tried something like this: {% for item in list_of_items %} {{item}}
 {% endfor %} Is it possible to pass a list from Python to JavaScript or should I pass each item from list one by one in a loop? How can I do this?

Output: This answer is unsafe, please refer to Mark Amery's answer below for the right way to do this. To pass some context data to javascript code, you have to serialize it in a way it will be "understood" by javascript (namely JSON). You also need to mark it as safe using the safe Jinja filter, to prevent your data from being htmlescaped. You can achieve this by doing something like that: The view import json @app.route('/') def my_view(): data = [1, 'foo'] return render_template('index.html', data=json.dumps(data)) The template <script type="text/javascript"> function test_func(data) { console.log(data); } test_func({{ data|safe }}) </script> Edit - exact answer So, to achieve exactly what you want (loop over a list of items, and pass them to a javascript function), you'd need to serialize every item in your list separately. Your code would then look like this: The view import json @app.route('/') def my_view(): data = [1, "foo"] return render_template('index.html', data=map(json.dumps, data)) The template {% for item in data %} <span onclick=someFunction({{ item|safe }});>{{ item }}</span> {% endfor %} Edit 2 In my example, I use Flask, I don't know what framework you're using, but you got the idea, you just have to make it fit the framework you use. Edit 3 (Security warning) NEVER EVER DO THIS WITH USER-SUPPLIED DATA, ONLY DO THIS WITH TRUSTED DATA! Otherwise, you would expose your application to XSS vulnerabilities!


Instruction:
Insert line at middle of file with Python?

Is there a way to do this? Say I have a file that's a list of names that goes like this: Alfred Bill Donald How could I insert the third name, "Charlie", at line x (in this case 3), and automatically send all others down one line? I've seen other questions like this, but they didn't get helpful answers. Can it be done, preferably with either a method or a loop?

Output: This is a way of doing the trick. with open("path_to_file", "r") as f: contents = f.readlines() contents.insert(index, value) with open("path_to_file", "w") as f: contents = "".join(contents) f.write(contents) index and value are the line and value of your choice, lines starting from 0.


Instruction:
How to find datetime 10 mins after current time?

I want to find out the datetime 10 mins after current time. Let's say we have from datetime import datetime now = datetime.now() new_now = datetime.strptime(now, '%a, %d %b %Y %H:%M:%S %Z') I want to find this now and new_now 10 minutes later. How can I do that?

Output: This is a duplicate of this question. You basically just need to add a timedelta of 10 minutes to get the time you want. from datetime import datetime, timedelta now = datetime.now() now_plus_10 = now + timedelta(minutes = 10)


Instruction:
Selenium Webdriver finding an element in a sub-element

I am trying to search for an element in a sub-element with Selenium (Version 2.28.0), but selenium does not seem to limit its search to the sub-element. Am I doing this wrong or is there a way to use element.find to search a sub-element? For an example I created a simple test webpage with this code: <!DOCTYPE html>

My First Heading

My first paragraph.

My Second Heading

My second paragraph.

My Third Heading

My third paragraph.

   My python (Version 2.6) code looks like this: from selenium import webdriver driver = webdriver.Firefox() # Open the test page with this instance of Firefox # element2 gets the second division as a web element element2 = driver.find_element_by_xpath("//div[@title='div2']") # Search second division for a paragraph with a class of 'test' and print the content print element2.find_element_by_xpath("//p[@class='test']").text # expected output: "My second paragraph." # actual output: "My first paragraph." If I run: print element2.get_attribute('innerHTML') It returns the html from the second division. So selenium is not limiting its search to element2. I would like to be able to find a sub-element of element2. This post suggests my code should work Selenium WebDriver access a sub element but his problem was caused by a time-out issue. Can anyone help me understand what is happening here?

Output: If you start an XPath expression with //, it begins searching from the root of document. To search relative to a particular element, you should prepend the expression with . instead: element2 = driver.find_element_by_xpath("//div[@title='div2']") element2.find_element_by_xpath(".//p[@class='test']").text


Instruction:
What does the delayed() function do (when used with joblib in Python)

I've read through the documentation, but I don't understand what is meant by: The delayed function is a simple trick to be able to create a tuple (function, args, kwargs) with a function-call syntax. I'm using it to iterate over the list I want to operate on (allImages) as follows: def joblib_loop(): Parallel(n_jobs=8)(delayed(getHog)(i) for i in allImages) This returns my HOG features, like I want (and with the speed gain using all my 8 cores), but I'm just not sure what it is actually doing. My Python knowledge is alright at best, and it's very possible that I'm missing something basic. Any pointers in the right direction would be most appreciated

Output: Perhaps things become clearer if we look at what would happen if instead we simply wrote Parallel(n_jobs=8)(getHog(i) for i in allImages) which, in this context, could be expressed more naturally as: Create a Parallel instance with n_jobs=8 create a generator for the list [getHog(i) for i in allImages] pass that generator to the Parallel instance What's the problem? By the time the list gets passed to the Parallel object, all getHog(i) calls have already returned - so there's nothing left to execute in Parallel! All the work was already done in the main thread, sequentially. What we actually want is to tell Python what functions we want to call with what arguments, without actually calling them - in other words, we want to delay the execution. This is what delayed conveniently allows us to do, with clear syntax. If we want to tell Python that we'd like to call foo(2, g=3) sometime later, we can simply write delayed(foo)(2, g=3). Returned is the tuple (foo, [2], {g: 3}), containing: a reference to the function we want to call, e.g.foo all arguments (short "args") without a keyword, e.g.t 2 all keyword arguments (short "kwargs"), e.g. g=3 So, by writing Parallel(n_jobs=8)(delayed(getHog)(i) for i in allImages), instead of the above sequence, now the following happens: A Parallel instance with n_jobs=8 gets created The list [delayed(getHog)(i) for i in allImages] gets created, evaluating to [(getHog, [img1], {}), (getHog, [img2], {}), ... ] That list is passed to the Parallel instance The Parallel instance creates 8 threads and distributes the tuples from the list to them Finally, each of those threads starts executing the tuples, i.e., they call the first element with the second and the third elements unpacked as arguments tup[0](*tup[1], **tup[2]), turning the tuple back into the call we actually intended to do, getHog(img2).


Instruction:
Move seaborn plot legend to a different position

I'm using factorplot(kind="bar") with seaborn. The plot is fine except the legend is misplaced: too much to the right, text goes out of the plot's shaded area. How do I make seaborn place the legend somewhere else, such as in top-left instead of middle-right?

Output: Building on @user308827's answer: you can use legend=False in factorplot and specify the legend through matplotlib: import seaborn as sns import matplotlib.pyplot as plt sns.set(style="whitegrid") titanic = sns.load_dataset("titanic") g = sns.factorplot("class", "survived", "sex", data=titanic, kind="bar", size=6, palette="muted", legend=False) g.despine(left=True) plt.legend(loc='upper left') g.set_ylabels("survival probability") plt acts on the current axes. To get axes from a FacetGrid use fig. g.fig.get_axes()[0].legend(loc='lower left')


Instruction:
How can I access global variable inside class in Python

I have this: g_c = 0 class TestClass(): global g_c def run(self): for i in range(10): g_c = 1 print(g_c) t = TestClass() t.run() print(g_c) how can I actually modify my global variable g_c?

Output: By declaring it global inside the function that accesses it: g_c = 0 class TestClass(): def run(self): global g_c for i in range(10): g_c = 1 print(g_c) The Python documentation says this, about the global statement: The global statement is a declaration which holds for the entire current code block.


Instruction:
Import arbitrary python source file. (Python 3.3+)

How can I import an arbitrary python source file (whose filename could contain any characters, and does not always ends with .py) in Python 3.3+? I used imp.load_module as follows: >>> import imp >>> path = '/tmp/a-b.txt' >>> with open(path, 'U') as f: ... mod = imp.load_module('a_b', f, path, ('.py', 'U', imp.PY_SOURCE)) ... >>> mod <module 'a_b' from '/tmp/a-b.txt'> It still works in Python 3.3, but according to imp.load_module documentation, it is deprecated: Deprecated since version 3.3: Unneeded as loaders should be used to load modules and find_module() is deprecated. and imp module documentation recommends to use importlib: Note New programs should use importlib rather than this module. What is the proper way to load an arbitrary python source file in Python 3.3+ without using the deprecated imp.load_module function?

Output: Found a solution from importlib test code. Using importlib.machinery.SourceFileLoader: >>> import importlib.machinery >>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt') >>> mod = loader.load_module() >>> mod <module 'a_b' from '/tmp/a-b.txt'> NOTE: only works in Python 3.3+. UPDATE Loader.load_module is deprecated since Python 3.4. Use Loader.exec_module instead: >>> import types >>> import importlib.machinery >>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt') >>> mod = types.ModuleType(loader.name) >>> loader.exec_module(mod) >>> mod <module 'a_b'> >>> import importlib.machinery >>> import importlib.util >>> loader = importlib.machinery.SourceFileLoader('a_b', '/tmp/a-b.txt') >>> spec = importlib.util.spec_from_loader(loader.name, loader) >>> mod = importlib.util.module_from_spec(spec) >>> loader.exec_module(mod) >>> mod <module 'a_b' from '/tmp/a-b.txt'>


Instruction:
Validating detailed types in Python dataclasses

Python 3.7 was released a while ago, and I wanted to test some of the fancy new dataclass+typing features. Getting hints to work right is easy enough, with both native types and those from the typing module: >>> import dataclasses >>> import typing as ty >>> ... @dataclasses.dataclass ... class Structure: ... a_str: str ... a_str_list: ty.List[str] ... >>> my_struct = Structure(a_str='test', a_str_list=['t', 'e', 's', 't']) >>> my_struct.a_str_list[0]. # IDE suggests all the string methods :) But one other thing that I wanted to try was forcing the type hints as conditions during runtime, i.e. it should not be possible for a dataclass with incorrect types to exist. It can be implemented nicely with __post_init__: >>> @dataclasses.dataclass ... class Structure: ... a_str: str ... a_str_list: ty.List[str] ... ... def validate(self): ... ret = True ... for field_name, field_def in self.__dataclass_fields__.items(): ... actual_type = type(getattr(self, field_name)) ... if actual_type != field_def.type: ... print(f"\t{field_name}: '{actual_type}' instead of '{field_def.type}'") ... ret = False ... return ret ... ... def __post_init__(self): ... if not self.validate(): ... raise ValueError('Wrong types') This kind of validate function works for native types and custom classes, but not those specified by the typing module: >>> my_struct = Structure(a_str='test', a_str_list=['t', 'e', 's', 't']) Traceback (most recent call last): a_str_list: '<class 'list'>' instead of 'typing.List[str]' ValueError: Wrong types Is there a better approach to validate an untyped list with a typing-typed one? Preferably one that doesn't include checking the types of all elements in any list, dict, tuple, or set that is a dataclass' attribute. Revisiting this question after a couple of years, I've now moved to use pydantic in cases where I want to validate classes that I'd normally just define a dataclass for. I'll leave my mark with the currently accepted answer though, since it correctly answers the original question and has outstanding educational value.

Output: Instead of checking for type equality, you should use isinstance. But you cannot use a parametrized generic type (typing.List[int]) to do so, you must use the "generic" version (typing.List). So you will be able to check for the container type but not the contained types. Parametrized generic types define an __origin__ attribute that you can use for that. Contrary to Python 3.6, in Python 3.7 most type hints have a useful __origin__ attribute. Compare: # Python 3.6 >>> import typing >>> typing.List.__origin__ >>> typing.List[int].__origin__ typing.List and # Python 3.7 >>> import typing >>> typing.List.__origin__ <class 'list'> >>> typing.List[int].__origin__ <class 'list'> Python 3.8 introduce even better support with the typing.get_origin() introspection function: # Python 3.8 >>> import typing >>> typing.get_origin(typing.List) <class 'list'> >>> typing.get_origin(typing.List[int]) <class 'list'> Notable exceptions being typing.Any, typing.Union and typing.ClassVar… Well, anything that is a typing._SpecialForm does not define __origin__. Fortunately: >>> isinstance(typing.Union, typing._SpecialForm) True >>> isinstance(typing.Union[int, str], typing._SpecialForm) False >>> typing.get_origin(typing.Union[int, str]) typing.Union But parametrized types define an __args__ attribute that store their parameters as a tuple; Python 3.8 introduce the typing.get_args() function to retrieve them: # Python 3.7 >>> typing.Union[int, str].__args__ (<class 'int'>, <class 'str'>) # Python 3.8 >>> typing.get_args(typing.Union[int, str]) (<class 'int'>, <class 'str'>) So we can improve type checking a bit: for field_name, field_def in self.__dataclass_fields__.items(): if isinstance(field_def.type, typing._SpecialForm): # No check for typing.Any, typing.Union, typing.ClassVar (without parameters) continue try: actual_type = field_def.type.__origin__ except AttributeError: # In case of non-typing types (such as <class 'int'>, for instance) actual_type = field_def.type # In Python 3.8 one would replace the try/except with # actual_type = typing.get_origin(field_def.type) or field_def.type if isinstance(actual_type, typing._SpecialForm): # case of typing.Union[…] or typing.ClassVar[…] actual_type = field_def.type.__args__ actual_value = getattr(self, field_name) if not isinstance(actual_value, actual_type): print(f"\t{field_name}: '{type(actual_value)}' instead of '{field_def.type}'") ret = False This is not perfect as it won't account for typing.ClassVar[typing.Union[int, str]] or typing.Optional[typing.List[int]] for instance, but it should get things started. Next is the way to apply this check. Instead of using __post_init__, I would go the decorator route: this could be used on anything with type hints, not only dataclasses: import inspect import typing from contextlib import suppress from functools import wraps def enforce_types(callable): spec = inspect.getfullargspec(callable) def check_types(*args, **kwargs): parameters = dict(zip(spec.args, args)) parameters.update(kwargs) for name, value in parameters.items(): with suppress(KeyError): # Assume un-annotated parameters can be any type type_hint = spec.annotations[name] if isinstance(type_hint, typing._SpecialForm): # No check for typing.Any, typing.Union, typing.ClassVar (without parameters) continue try: actual_type = type_hint.__origin__ except AttributeError: # In case of non-typing types (such as <class 'int'>, for instance) actual_type = type_hint # In Python 3.8 one would replace the try/except with # actual_type = typing.get_origin(type_hint) or type_hint if isinstance(actual_type, typing._SpecialForm): # case of typing.Union[…] or typing.ClassVar[…] actual_type = type_hint.__args__ if not isinstance(value, actual_type): raise TypeError('Unexpected type for \'{}\' (expected {} but found {})'.format(name, type_hint, type(value))) def decorate(func): @wraps(func) def wrapper(*args, **kwargs): check_types(*args, **kwargs) return func(*args, **kwargs) return wrapper if inspect.isclass(callable): callable.__init__ = decorate(callable.__init__) return callable return decorate(callable) Usage being: @enforce_types @dataclasses.dataclass class Point: x: float y: float @enforce_types def foo(bar: typing.Union[int, str]): pass Appart from validating some type hints as suggested in the previous section, this approach still have some drawbacks: type hints using strings (class Foo: def __init__(self: 'Foo'): pass) are not taken into account by inspect.getfullargspec: you may want to use typing.get_type_hints and inspect.signature instead; a default value which is not the appropriate type is not validated: @enforce_type def foo(bar: int = None): pass foo() does not raise any TypeError. You may want to use inspect.Signature.bind in conjuction with inspect.BoundArguments.apply_defaults if you want to account for that (and thus forcing you to define def foo(bar: typing.Optional[int] = None)); variable number of arguments can't be validated as you would have to define something like def foo(*args: typing.Sequence, **kwargs: typing.Mapping) and, as said at the beginning, we can only validate containers and not contained objects. Update After this answer got some popularity and a library heavily inspired by it got released, the need to lift the shortcomings mentioned above is becoming a reality. So I played a bit more with the typing module and will propose a few findings and a new approach here. For starter, typing is doing a great job in finding when an argument is optional: >>> def foo(a: int, b: str, c: typing.List[str] = None): ... pass ... >>> typing.get_type_hints(foo) {'a': <class 'int'>, 'b': <class 'str'>, 'c': typing.Union[typing.List[str], NoneType]} This is pretty neat and definitely an improvement over inspect.getfullargspec, so better use that instead as it can also properly handle strings as type hints. But typing.get_type_hints will bail out for other kind of default values: >>> def foo(a: int, b: str, c: typing.List[str] = 3): ... pass ... >>> typing.get_type_hints(foo) {'a': <class 'int'>, 'b': <class 'str'>, 'c': typing.List[str]} So you may still need extra strict checking, even though such cases feels very fishy. Next is the case of typing hints used as arguments for typing._SpecialForm, such as typing.Optional[typing.List[str]] or typing.Final[typing.Union[typing.Sequence, typing.Mapping]]. Since the __args__ of these typing._SpecialForms is always a tuple, it is possible to recursively find the __origin__ of the hints contained in that tuple. Combined with the above checks, we will then need to filter any typing._SpecialForm left. Proposed improvements: import inspect import typing from functools import wraps def _find_type_origin(type_hint): if isinstance(type_hint, typing._SpecialForm): # case of typing.Any, typing.ClassVar, typing.Final, typing.Literal, # typing.NoReturn, typing.Optional, or typing.Union without parameters return actual_type = typing.get_origin(type_hint) or type_hint # requires Python 3.8 if isinstance(actual_type, typing._SpecialForm): # case of typing.Union[…] or typing.ClassVar[…] or … for origins in map(_find_type_origin, typing.get_args(type_hint)): yield from origins else: yield actual_type def _check_types(parameters, hints): for name, value in parameters.items(): type_hint = hints.get(name, typing.Any) actual_types = tuple(_find_type_origin(type_hint)) if actual_types and not isinstance(value, actual_types): raise TypeError( f"Expected type '{type_hint}' for argument '{name}'" f" but received type '{type(value)}' instead" ) def enforce_types(callable): def decorate(func): hints = typing.get_type_hints(func) signature = inspect.signature(func) @wraps(func) def wrapper(*args, **kwargs): parameters = dict(zip(signature.parameters, args)) parameters.update(kwargs) _check_types(parameters, hints) return func(*args, **kwargs) return wrapper if inspect.isclass(callable): callable.__init__ = decorate(callable.__init__) return callable return decorate(callable) def enforce_strict_types(callable): def decorate(func): hints = typing.get_type_hints(func) signature = inspect.signature(func) @wraps(func) def wrapper(*args, **kwargs): bound = signature.bind(*args, **kwargs) bound.apply_defaults() parameters = dict(zip(signature.parameters, bound.args)) parameters.update(bound.kwargs) _check_types(parameters, hints) return func(*args, **kwargs) return wrapper if inspect.isclass(callable): callable.__init__ = decorate(callable.__init__) return callable return decorate(callable) Thanks to @Aran-Fey that helped me improve this answer.


Instruction:
python typing signature (typing.Callable) for function with kwargs

I heavily use python typing support from python 3. Recently I was trying to pass a function as an argument and I do not find any help for using kwargs in typing.Callable signature. Please check the code below and the comments. import typing # some function with a type signature def fn1_as_arg_with_kwargs(a: int, b: float) -> float: return a + b # some function with a type signature def fn2_as_arg_with_kwargs(a: int, b: float) -> float: return a * b # function that get callables as arg # this works with typing def function_executor( a: int, b: float, fn: typing.Callable[[int, float], float]): return fn(a, b) # But what if I want to name my kwargs # (something like below which does not work) # ... this will help me more complex scenarios # ... or am I expecting a lot from python3 ;) def function_executor( a: int, b: float, fn: typing.Callable[["a": int, "b": float], float]): return fn(a=a, b=b)

Output: You are probably looking for Callback protocols. In short, when you want to express a callable with a complex signature, what you'll want to do is to create a custom Protocol that defines a __call__ method with the precise signature you want. For example, in your case: from typing import Protocol # Or, if you want to support Python 3.7 and below, install the typing_extensions # module via pip and do the below: from typing_extensions import Protocol class MyCallable(Protocol): def __call__(self, a: int, b: float) -> float: ... def good(a: int, b: float) -> float: ... def bad(x: int, y: float) -> float: ... def function_executor(a: int, b: float, fn: MyCallable) -> float: return fn(a=a, b=b) function_executor(1, 2.3, good) # Ok! function_executor(1, 2.3, bad) # Errors If you try type-checking this program using mypy, you'll get the following (admittedly cryptic) error on the last line: Argument 3 to "function_executor" has incompatible type "Callable[[int, float], float]"; expected "MyCallable" (Callback protocols are somewhat new, so hopefully the quality of the error messages will improve over time.)


Instruction:
What is validation data used for in a Keras Sequential model?

My question is simple, what is the validation data passed to model.fit in a Sequential model used for? And, does it affect how the model is trained (normally a validation set is used, for example, to choose hyper-parameters in a model, but I think this does not happen here)? I am talking about the validation set that can be passed like this: # Create model model = Sequential() # Add layers model.add(...) # Train model (use 10% of training set as validation set) history = model.fit(X_train, Y_train, validation_split=0.1) # Train model (use validation data as validation set) history = model.fit(X_train, Y_train, validation_data=(X_test, Y_test)) I investigated a bit, and I saw that keras.models.Sequential.fit calls keras.models.training.fit, which creates variables like val_accand val_loss (which can be accessed from Callbacks). keras.models.training.fit also calls keras.models.training._fit_loop, which adds the validation data to the callbacks.validation_data, and also calls keras.models.training._test_loop, which will loop the validation data in batches on the self.test_function of the model. The result of this function is used to fill the values of the logs, which are the values accessible from the callbacks. After seeing all this, I feel that the validation set passed to model.fit is not used to validate anything during training, and its only use is to get feedback on how the trained model will perform in every epoch for a completely independent set. Therefore, it would be okey to use the same validation and test set, right? Could anyone confirm if the validation set in model.fit has any other goal besides being read from the callbacks?

Output: If you want to build a solid model you have to follow that specific protocol of splitting your data into three sets: One for training, one for validation and one for final evaluation, which is the test set. The idea is that you train on your training data and tune your model with the results of metrics (accuracy, loss etc) that you get from your validation set. Your model doesn't "see" your validation set and isn't in any way trained on it, but you as the architect and master of the hyperparameters tune the model according to this data. Therefore it indirectly influences your model because it directly influences your design decisions. You nudge your model to work well with the validation data and that can possibly bring in a tilt. Exactly that is the reason you only evaluate your model's final score on data that neither your model nor you yourself has used – and that is the third chunk of data, your test set. Only this procedure makes sure you get an unaffected view of your models quality and ability to generalize what is has learned on totally unseen data.


Instruction:
Accessing Pandas column using squared brackets vs using a dot (like an attribute)

In both the bellow cases: import pandas d = {'col1': 2, 'col2': 2.5} df = pandas.DataFrame(data=d, index=[0]) print(df['col2']) print(df.col2) Both methods can be used to index on a column and yield the same result, so is there any difference between them?

Output: The "dot notation", i.e. df.col2 is the attribute access that's exposed as a convenience. You may access an index on a Series, column on a DataFrame, and an item on a Panel directly as an attribute: df['col2'] does the same: it returns a pd.Series of the column. A few caveats about attribute access: you cannot add a column (df.new_col = x won't work, worse: it will silently actually create a new attribute rather than a column - think monkey-patching here) it won't work if you have spaces in the column name or if the column name is an integer.


Instruction:
Difference between class and instance methods

I was reading PEP 8 (style guide), and I noticed that it suggested to use self as the first argument in an instance method, but cls as the first argument in a class method. I've used and written a few classes, but I've never encountered a class method (well, a method which passes cls as a parameter). What are some examples?

Output: Instance methods When creating an instance method, the first parameter is always self. You can name it anything you want, but the meaning will always be the same, and you should use self since it's the naming convention. self is (usually) passed hiddenly when calling an instance method; it represents the instance calling the method. Here's an example of a class called Inst that has an instance method called introduce(): class Inst: def __init__(self, name): self.name = name def introduce(self): print("Hello, I am %s, and my name is " %(self, self.name)) Now to call this method, we first need to create an instance of our class. Once we have an instance, we can call introduce() on it, and the instance will automatically be passed as self: myinst = Inst("Test Instance") otherinst = Inst("An other instance") myinst.introduce() # outputs: Hello, I am <Inst object at x>, and my name is Test Instance otherinst.introduce() # outputs: Hello, I am <Inst object at y>, and my name is An other instance As you see, we're not passing the parameter self. It gets hiddenly passed with the period operator. We're calling Inst class's instance method introduce, with the parameter of myinst or otherinst. This means that we can call Inst.introduce(myinst) and get the exact same result. Class methods The idea of a class method is very similar to an instance method, only difference being that instead of passing the instance hiddenly as a first parameter, we're now passing the class itself as a first parameter. class Cls: @classmethod def introduce(cls): print("Hello, I am %s!" %cls) Since we're passing only a class to the method, no instance is involved. This means that we don't need an instance at all, and we call the class method as if it was a static function: Cls.introduce() # same as Cls.introduce(Cls) # outputs: Hello, I am <class 'Cls'> Notice that again Cls is passed hiddenly, so we could also say Cls.introduce(Inst) and get output "Hello, I am <class 'Inst'>. This is particularly useful when we're inheriting a class from Cls: class SubCls(Cls): pass SubCls.introduce() # outputs: Hello, I am <class 'SubCls'>


Instruction:
What do backticks mean to the Python interpreter? Example: `num`

I'm playing around with list comprehensions and I came across this little snippet on another site: return ''.join([`num` for num in xrange(loop_count)]) I spent a few minutes trying to replicate the function (by typing) before realising the `num` bit was breaking it. What does enclosing a statement in those characters do? From what I can see it is the equivalent of str(num). But when I timed it: return ''.join([str(num) for num in xrange(10000000)]) It takes 4.09 seconds whereas: return ''.join([`num` for num in xrange(10000000)]) takes 2.43 seconds. Both give identical results, but one is a lot slower. What is going on here? Oddly... repr() gives slightly slower results than `num`. 2.99 seconds vs 2.43 seconds. I am using Python 2.6 (haven't tried 3.0 yet).

Output: Backticks are a deprecated alias for repr(). Don't use them any more; the syntax was removed in Python 3.0. Using backticks seems to be faster than using repr(num) or num.__repr__() in version 2.x. I guess it's because additional dictionary lookup is required in the global namespace (for repr), or in the object's namespace (for __repr__), respectively. Using the dis module proves my assumption: def f1(a): return repr(a) def f2(a): return a.__repr__() def f3(a): return `a` Disassembling shows: >>> import dis >>> dis.dis(f1) 3 0 LOAD_GLOBAL 0 (repr) 3 LOAD_FAST 0 (a) 6 CALL_FUNCTION 1 9 RETURN_VALUE >>> dis.dis(f2) 6 0 LOAD_FAST 0 (a) 3 LOAD_ATTR 0 (__repr__) 6 CALL_FUNCTION 0 9 RETURN_VALUE >>> dis.dis(f3) 9 0 LOAD_FAST 0 (a) 3 UNARY_CONVERT 4 RETURN_VALUE f1 involves a global lookup for repr, f2 an attribute lookup for __repr__, whereas the backtick operator is implemented in a separate opcode. Since there is no overhead for dictionary lookup (LOAD_GLOBAL/LOAD_ATTR) nor for function calls (CALL_FUNCTION), backticks are faster. I guess that the Python folks decided that having a separate low-level operation for repr() is not worth it, and having both repr() and backticks violates the principle "There should be one-- and preferably only one --obvious way to do it" so the feature was removed in Python 3.0.


Instruction:
Why can't I iterate twice over the same iterator? How can I "reset" the iterator or reuse the data?

Consider the code: def test(data): for row in data: print("first loop") for row in data: print("second loop") When data is an iterator, for example a list iterator or a generator expression*, this does not work: >>> test(iter([1, 2])) first loop first loop >>> test((_ for _ in [1, 2])) first loop first loop This prints first loop a few times, since data is non-empty. However, it does not print second loop. Why does iterating over data work the first time, but not the second time? How can I make it work a second time? Aside from for loops, the same problem appears to occur with any kind of iteration: list/set/dict comprehensions, passing the iterator to list(), sum() or reduce(), etc. On the other hand, if data is another kind of iterable, such as a list or a range (which are both sequences), both loops run as expected: >>> test([1, 2]) first loop first loop second loop second loop >>> test(range(2)) first loop first loop second loop second loop * More examples: file objects generators created from an explicit generator function filter, map, and zip objects (in 3.x) enumerate objects csv.readers various iterators defined in the itertools standard library For general theory and terminology explanation, see What are iterator, iterable, and iteration?. To detect whether the input is an iterator or a "reusable" iterable, see Ensure that an argument can be iterated twice.

Output: An iterator can only be consumed once. For example: data = [1, 2, 3] it = iter(data) next(it) # => 1 next(it) # => 2 next(it) # => 3 next(it) # => StopIteration When the iterator is supplied to a for loop instead, that last StopIteration will cause it to exit the first time. Trying to use the same iterator in another for loop will cause StopIteration again immediately, because the iterator has already been consumed. A simple way to work around this is to save all the elements to a list, which can be traversed as many times as needed. For example: data = list(it) If the iterator would iterate over many elements at roughly the same time, however, it's a better idea to create independent iterators using tee(): import itertools it1, it2 = itertools.tee(data, 2) # create as many as needed Now each one can be iterated over separately: next(it1) # => 1 next(it1) # => 2 next(it2) # => 1 next(it2) # => 2 next(it1) # => 3 next(it2) # => 3


Instruction:
What is the difference between pickle and shelve?

When is it appropriate to use pickle, and when is it appropriate to use shelve? That is to say, what do they do differently from each other? From my research, I understood that pickle can turn every Python object into stream of bytes which can be persisted into a file. Then why do we need shelve as well? Isn't pickle faster?

Output: pickle is for serializing some object (or objects) as a single bytestream in a file. shelve builds on top of pickle and implements a serialization dictionary where objects are pickled, but associated with a key (some string), so you can load your shelved data file and access your pickled objects via keys. This could be more convenient were you to be serializing many objects. Here is an example of usage between the two. (should work in latest versions of Python 2.7 and Python 3.x). pickle Example import pickle integers = [1, 2, 3, 4, 5] with open('pickle-example.p', 'wb') as pfile: pickle.dump(integers, pfile) This will dump the integers list to a binary file called pickle-example.p. Now try reading the pickled file back. import pickle with open('pickle-example.p', 'rb') as pfile: integers = pickle.load(pfile) print(integers) The above should output [1, 2, 3, 4, 5]. shelve Example import shelve integers = [1, 2, 3, 4, 5] # If you're using Python 2.7, import contextlib and use # the line: # with contextlib.closing(shelve.open('shelf-example', 'c')) as shelf: with shelve.open('shelf-example', 'c') as shelf: shelf['ints'] = integers Notice how you add objects to the shelf via dictionary-like access. Read the object back in with code like the following: import shelve # If you're using Python 2.7, import contextlib and use # the line: # with contextlib.closing(shelve.open('shelf-example', 'r')) as shelf: with shelve.open('shelf-example', 'r') as shelf: for key in shelf.keys(): print(repr(key), repr(shelf[key])) The output will be 'ints', [1, 2, 3, 4, 5].


Instruction:
Multiple top-level packages discovered in a flat-layout

I am trying to install a library from the source that makes use of Poetry, but I get this error error: Multiple top-level packages discovered in a flat-layout: ['tulips', 'fixtures']. To avoid accidental inclusion of unwanted files or directories, setuptools will not proceed with this build. If you are trying to create a single distribution with multiple packages on purpose, you should not rely on automatic discovery. Instead, consider the following options: 1. set up custom discovery (`find` directive with `include` or `exclude`) 2. use a `src-layout` 3. explicitly set `py_modules` or `packages` with a list of names To find more information, look for "package discovery" on setuptools docs What do I need to do to fix it?

Output: Based on this comment on a GitHub issue, adding the following lines to your pyproject.toml might solve your problem: [tool.setuptools] py-modules = [] (For my case, the other workaround provided in that comment, i.e. adding py_modules=[] as a keyword argument to the setup() function in setup.py worked) See Package Discovery and Namespace Packages for additional information.


Instruction:
Paramiko : Error reading SSH protocol banner

Recently, I made a code that connect to work station with different usernames (thanks to a private key) based on paramiko. I never had any issues with it, but today, I have that : SSHException: Error reading SSH protocol banner This is strange because it happens randomly on any connections. Is there any way to fix it ?

Output: It depends on what you mean by "fix". The underlying cause, as pointed out in the comments, are congestion/lack of resources. In that way, it's similar to some HTTP codes. That's the normal cause, it could be that the ssh server is returning the wrong header data. 429 Too Many Requests, tells the client to use rate limiting, or sometimes APIs will return 503 in a similar way, if you exceed your quota. The idea being, to try again later, with a delay. You can attempt to handle this exception in your code, wait a little while, and try again. You can also edit your transport.py file, to set the banner timeout to something higher. If you have an application where it doesn't matter how quickly the server responds, you could set this to 60 seconds. EDIT: Editing your transport file is no longer needed as per Greg's answer. When you call connect, you can pass a banner_timeout (which solves this issue), a timeout (for the underlying TCP), and an auth_timeout (waiting for authentication response). Greg's answer has a code example with banner_timeout that you can directly lift.


Instruction:
Django Admin Show Image from Imagefield

While I can show an uploaded image in list_display, is it possible to do this on the model page (the page you get for changing an element)? A quick sample model would be: class Model1(models.Model): image = models.ImageField(upload_to=directory) The default admin shows the URL of the uploaded image but not the image itself. Thanks!

Output: In addition to the answer of Michael C. O'Connor Note that since Django v.1.9 (updated - tested and worked all the way to Django 3.0) image_tag.allow_tags = True is deprecated and you should use format_html(), format_html_join(), or mark_safe() instead So if you are storing your uploaded files in your public /directory folder, your code should look like this: from django.utils.html import mark_safe Class Model1(models.Model): image = models.ImageField(upload_to=directory) def image_tag(self): return mark_safe('<img src="/directory/%s" width="150" height="150" />' % (self.image)) image_tag.short_description = 'Image' and in your admin.py add: fields = ['image_tag'] readonly_fields = ['image_tag']


Instruction:
Django 1.7 - "No migrations to apply" when run migrate after makemigrations

I use Django1.7 with Mezzanine. I create simple profile (according to Mezzanine documentation) stored in separate app "profiles": class RoadmapProfile(models.Model): user = models.OneToOneField("auth.User") fullname = models.CharField(max_length=100, verbose_name="Full name") Creation of migrations returns: Migrations for 'profiles': 0001_initial.py: - Create model RoadmapProfile When I run "migrate profiles": Operations to perform: Apply all migrations: profiles Running migrations: No migrations to apply. The issue is, when I try to open any page related to mezzanine.accounts (for example update account), it crashes with: OperationalError at /accounts/update/ no such column: profiles_roadmapprofile.fullname What I have done wrong?

Output: Sounds like your initial migration was faked because the table already existed (probably with an outdated schema): https://docs.djangoproject.com/en/1.8/topics/migrations/#adding-migrations-to-apps "This will make a new initial migration for your app. Now, when you run migrate, Django will detect that you have an initial migration and that the tables it wants to create already exist, and will mark the migration as already applied." Otherwise you would get an no-such-table error :) Did you clean up the applied-migrations table? That's also a common cause for non-applied migrations.


Instruction:
How to remove a path prefix in python?

I wanted to know what is the pythonic function for this : I want to remove everything before the wa path. p = path.split('/') counter = 0 while True: if p[counter] == 'wa': break counter += 1 path = '/'+'/'.join(p[counter:]) For instance, I want '/book/html/wa/foo/bar/' to become '/wa/foo/bar/'.

Output: A better answer would be to use os.path.relpath: http://docs.python.org/3/library/os.path.html#os.path.relpath >>> import os >>> full_path = '/book/html/wa/foo/bar/' >>> relative_path = '/book/html' >>> print(os.path.relpath(full_path, relative_path)) 'wa/foo/bar'


Instruction:
Plotting histograms from grouped data in a pandas DataFrame

How do I plot a block of histograms from a group of data in a dataframe? For example, given: from pandas import DataFrame import numpy as np x = ['A']*300 + ['B']*400 + ['C']*300 y = np.random.randn(1000) df = DataFrame({'Letter': x, 'N': y}) I tried: df.groupby('Letter').hist() ...which failed with the error message: TypeError: cannot concatenate 'str' and 'float' objects

Output: I'm on a roll, just found an even simpler way to do it using the by keyword in the hist method: df.hist('N', by='Letter') That's a very handy little shortcut for quickly scanning your grouped data! For future visitors, the product of this call is the following chart: In answer to questions below, here's an example of specific tailoring of the histogram plots: # import libraries import pandas as pd import numpy as np # Create test dataframe x = ['A']*300 + ['B']*400 + ['C']*300 y = np.random.randn(1000) z = np.random.randn(1000) df = pd.DataFrame({'Letter':x, 'N1':y, 'N2':z}) # Plot histograms axes = df.hist(['N1','N2'], by='Letter',bins=10, layout=(2,2), legend=True, yrot=90,sharex=True,sharey=True, log=True, figsize=(6,6)) for ax in axes.flatten(): ax.set_xlabel('N') ax.set_ylabel('Count') ax.set_ylim(bottom=1,top=100)


Instruction:
Using a dictionary to select function to execute

I am trying to use functional programming to create a dictionary containing a key and a function to execute: myDict={} myItems=("P1","P2","P3",...."Pn") def myMain(key): def ExecP1(): pass def ExecP2(): pass def ExecP3(): pass ... def ExecPn(): pass Now, I have seen a code used to find the defined functions in a module, and I need to do something like this: for myitem in myItems: myDict[myitem] = ??? #to dynamically find the corresponding function So my question is, How do I make a list of all the Exec functions and then assign them to the desired item using the a dictionary? so at the end I will have myDict["P1"]() #this will call ExecP1() My real problem is that I have tons of those items and I making a library that will handle them so the final user only needs to call myMain("P1") I think using the inspect module, but I am not so sure how to do it. My reason to avoid: def ExecPn(): pass myDict["Pn"]=ExecPn is that I have to protect code as I am using it to provide a scripting feature within my application.

Output: Not proud of it, but: def myMain(key): def ExecP1(): pass def ExecP2(): pass def ExecP3(): pass def ExecPn(): pass locals()['Exec' + key]() I do however recommend that you put those in a module/class whatever, this is truly horrible. If you are willing to add a decorator for each function, you can define a decorator which adds each function to a dictionary: def myMain(key): tasks = {} def task(task_fn): tasks[task_fn.__name__] = task_fn @task def ExecP1(): print(1) @task def ExecP2(): print(2) @task def ExecP3(): print(3) @task def ExecPn(): print(4) tasks['Exec' + key]() Another option is to place all the functions under a class (or in a different module) and use getattr: def myMain(key): class Tasks: def ExecP1(): print(1) def ExecP2(): print(2) def ExecP3(): print(3) def ExecPn(): print(4) task = getattr(Tasks, 'Exec' + key) task()


Instruction:
Find out if a matrix is positive definite with NumPy

How can I find out if a matrix is positive definite? My matrix is a NumPy matrix. I was expecting to find any related method in the NumPy library, but I didn't have any success.

Output: You can also check if all the eigenvalues of matrix are positive. If so, the matrix is positive definite: import numpy as np def is_pos_def(x): return np.all(np.linalg.eigvals(x) > 0)


Instruction:
Truncate to three decimals in Python

How do I get 1324343032.324? As you can see below, the following do not work: >>1324343032.324325235 * 1000 / 1000 1324343032.3243253 >>int(1324343032.324325235 * 1000) / 1000.0 1324343032.3239999 >>round(int(1324343032.324325235 * 1000) / 1000.0,3) 1324343032.3239999 >>str(1324343032.3239999) '1324343032.32'

Output: You can use an additional float() around it if you want to preserve it as a float. val = '%.3f'%(1324343032.324325235)


Instruction:
Using quotation marks inside quotation marks

When I want to do a print command in Python and I need to use quotation marks, I don't know how to do it without closing the string. For instance: print " "a word that needs quotation marks" " But when I try to do what I did above, I end up closing the string and I can't put the word I need between quotation marks. How can I do that?

Output: You could do this in one of three ways: Use single and double quotes together: print('"A word that needs quotation marks"') "A word that needs quotation marks" Escape the double quotes within the string: print("\"A word that needs quotation marks\"") "A word that needs quotation marks" Use triple-quoted strings: print(""" "A word that needs quotation marks" """) "A word that needs quotation marks"


Instruction:
Why do I get a "ModuleNotFoundError" in VS Code despite the fact that I already installed the module?

I'm trying to debug some Python code using VS Code. I'm getting the following error about a module that I am sure is already installed. Exception has occurred: ModuleNotFoundError No module named 'SimpleITK' File "C:\Users\Mido\Desktop\ProstateX-project\src\01-preprocessing\03_resample_nifti.py", line 8, in <module> import SimpleITK as sitk I installed the module using sudo pip install SimpleITK I know that it was installed because I was getting a similar error when I ran the code through the command line, and it was fixed by doing the above. I don't understand why VS Code does not recognize that it's already installed.

Output: sudo pip install is most likely installing globally into a Python interpreter that is different than the one that you have selected in VS Code. Please select the Python interpreter you want to use and then install explicitly using that interpreter (if you're not using a virtual environment then use something like /path/to/python -m pip install SimpleITK, although I strongly recommend using a virtual environment and to not install packages globally).


Instruction:
drop into python interpreter while executing function

i have a python module with a function: def do_stuff(param1 = 'a'): if type(param1) == int: # enter python interpreter here do_something() else: do_something_else() is there a way to drop into the command line interpreter where i have the comment? so that if i run the following in python: >>> import my_module >>> do_stuff(1) i get my next prompt in the scope and context of where i have the comment in do_stuff()?

Output: Update for 2023: breakpoint() is included since py3.7 and will drop you into the debugger (pdb by default). If you have other debuggers installed (ipdb, pdbpp, ... to name a few) then breakpoint() will instead drop you into those. Please see these docs for more info: https://docs.python.org/3/library/pdb.html


Instruction:
django - comparing old and new field value before saving

I have a django model, and I need to compare old and new values of field BEFORE saving. I've tried the save() inheritance, and pre_save signal. It was triggered correctly, but I can't find the list of actually changed fields and can't compare old and new values. Is there a way? I need it for optimization of pre-save actions. Thank you!

Output: There is very simple django way for doing it. "Memorise" the values in model init like this: def __init__(self, *args, **kwargs): super(MyClass, self).__init__(*args, **kwargs) self.initial_parametername = self.parametername --- self.initial_parameternameX = self.parameternameX Real life example: At class: def __init__(self, *args, **kwargs): super(MyClass, self).__init__(*args, **kwargs) self.__important_fields = ['target_type', 'target_id', 'target_object', 'number', 'chain', 'expiration_date'] for field in self.__important_fields: setattr(self, '__original_%s' % field, getattr(self, field)) def has_changed(self): for field in self.__important_fields: orig = '__original_%s' % field if getattr(self, orig) != getattr(self, field): return True return False And then in modelform save method: def save(self, force_insert=False, force_update=False, commit=True): # Prep the data obj = super(MyClassForm, self).save(commit=False) if obj.has_changed(): # If we're down with commitment, save this shit if commit: obj.save(force_insert=True) return obj Edit ( 06.05.2024 ): Please read through all the comments that have been added since I added this post 10 years ago. Things have changed, and though even I still use this approach myself and recommend using it, there are cases, where you need to tweak the approach or forgo using it. Again, read through the comments others have added.


Instruction:
Correlation heatmap

I want to represent correlation matrix using a heatmap. There is something called correlogram in R, but I don't think there's such a thing in Python. How can I do this? The values go from -1 to 1, for example: [[ 1. 0.00279981 0.95173379 0.02486161 -0.00324926 -0.00432099] [ 0.00279981 1. 0.17728303 0.64425774 0.30735071 0.37379443] [ 0.95173379 0.17728303 1. 0.27072266 0.02549031 0.03324756] [ 0.02486161 0.64425774 0.27072266 1. 0.18336236 0.18913512] [-0.00324926 0.30735071 0.02549031 0.18336236 1. 0.77678274] [-0.00432099 0.37379443 0.03324756 0.18913512 0.77678274 1. ]] I was able to produce the following heatmap based on another question, but the problem is that my values get 'cut' at 0, so I would like to have a map which goes from blue(-1) to red(1), or something like that, but here values below 0 are not presented in an adequate way. Here's the code for that: plt.imshow(correlation_matrix,cmap='hot',interpolation='nearest')

Output: Another alternative is to use the heatmap function in seaborn to plot the covariance. This example uses the 'mpg' data set from seaborn. import seaborn as sns %matplotlib inline # load the Auto dataset auto_df = sns.load_dataset('mpg') # calculate the correlation matrix on the numeric columns corr = auto_df.select_dtypes('number').corr() # plot the heatmap sns.heatmap(corr) If you wanted to be even more fancy, you can use Pandas Style, for example: cmap = sns.diverging_palette(5, 250, as_cmap=True) def magnify(): return [dict(selector="th", props=[("font-size", "7pt")]), dict(selector="td", props=[('padding', "0em 0em")]), dict(selector="th:hover", props=[("font-size", "12pt")]), dict(selector="tr:hover td:hover", props=[('max-width', '200px'), ('font-size', '12pt')]) ] corr.style.background_gradient(cmap, axis=1)\ .format(precision=3)\ .set_properties(**{'max-width': '80px', 'font-size': '10pt'})\ .set_caption("Hover to magify")\ .set_table_styles(magnify())


Instruction:
Populating a dictionary using two for loops in Python

I'm trying to create a dictionary using two for loops. Here is my code: dicts = {} keys = range(4) values = ["Hi", "I", "am", "John"] for i in keys: for x in values: dicts[i] = x print(dicts) This outputs: {0: 'John', 1: 'John', 2: 'John', 3: 'John'} Why? I was planning on making it output: {0: 'Hi', 1: 'I', 2: 'am', 3: 'John'} Why doesn't it output that way and how do we make it output correctly?

Output: dicts = {} keys = range(4) values = ["Hi", "I", "am", "John"] for i in keys: dicts[i] = values[i] print(dicts) alternatively In [7]: dict(list(enumerate(values))) Out[7]: {0: 'Hi', 1: 'I', 2: 'am', 3: 'John'}


Instruction:
How do I catch a specific HTTP error in Python?

I have import urllib2 try: urllib2.urlopen("some url") except urllib2.HTTPError: <whatever> but what I end up is catching any kind of HTTP error. I want to catch only if the specified webpage doesn't exist (404?).

Output: Python 3 from urllib.error import HTTPError Python 2 from urllib2 import HTTPError Just catch HTTPError, handle it, and if it's not Error 404, simply use raise to re-raise the exception. See the Python tutorial. Here is a complete example for Python 2: import urllib2 from urllib2 import HTTPError try: urllib2.urlopen("some url") except HTTPError as err: if err.code == 404: <whatever> else: raise


Instruction:
How to wrap code/text in Jupyter notebooks

I am using jupyter-notebooks for python coding. Is there a way to wrap text/code in a jupyter notebook code cell? Picture provided below. By wrap text means "how text is wrapped in MS-word"

Output: When using jupyter-lab (rather than jupyter notebook) the solution is much simpler. You can do: Settings > Advanced setting Editor > TextEditor > checkbox enable Line Wrap.


Instruction:
Convert python datetime to timestamp in milliseconds

How do I convert a human-readable time such as 20.12.2016 09:38:42,76 to a Unix timestamp in milliseconds?

Output: In Python 3 this can be done in 2 steps: Convert timestring to datetime object Multiply the timestamp of the datetime object by 1000 to convert it to milliseconds. For example like this: from datetime import datetime dt_obj = datetime.strptime('20.12.2016 09:38:42,76', '%d.%m.%Y %H:%M:%S,%f') millisec = dt_obj.timestamp() * 1000 print(millisec) Output: 1482223122760.0 strptime accepts your timestring and a format string as input. The timestring (first argument) specifies what you actually want to convert to a datetime object. The format string (second argument) specifies the actual format of the string that you have passed. Here is the explanation of the format specifiers from the official documentation: %d - Day of the month as a zero-padded decimal number. %m - Month as a zero-padded decimal number. %Y - Year with century as a decimal number %H - Hour (24-hour clock) as a zero-padded decimal number. %M - Minute as a zero-padded decimal number. %S - Second as a zero-padded decimal number. %f - Microsecond as a decimal number, zero-padded to 6 digits.


Instruction:
Adding a scrollbar to a group of widgets in Tkinter

I am using Python to parse entries from a log file, and display the entry contents using Tkinter and so far it's been excellent. The output is a grid of label widgets, but sometimes there are more rows than can be displayed on the screen. I'd like to add a scrollbar, which looks like it should be very easy, but I can't figure it out. The documentation implies that only the List, Textbox, Canvas and Entry widgets support the scrollbar interface. None of these appear to be suitable for displaying a grid of widgets. It's possible to put arbitrary widgets in a Canvas widget, but you appear to have to use absolute co-ordinates, so I wouldn't be able to use the grid layout manager? I've tried putting the widget grid into a Frame, but that doesn't seem to support the scrollbar interface, so this doesn't work: mainframe = Frame(root, yscrollcommand=scrollbar.set) Can anyone suggest a way round this limitation? I'd hate to have to rewrite in PyQt and increase my executable image size by so much, just to add a scrollbar!

Output: Overview You can only associate scrollbars with a few widgets, and the root widget and Frame aren't part of that group of widgets. There are at least a couple of ways to do this. If you need a simple vertical or horizontal group of widgets, you can use a text widget and the window_create method to add widgets. This method is simple, but doesn't allow for a complex layout of the widgets. A more common general-purpose solution is to create a canvas widget and associate the scrollbars with that widget. Then, into that canvas embed the frame that contains your label widgets. Determine the width/height of the frame and feed that into the canvas scrollregion option so that the scrollregion exactly matches the size of the frame. Why put the widgets in a frame rather than directly in the canvas? A scrollbar attached to a canvas can only scroll items created with one of the create_ methods. You cannot scroll items added to a canvas with pack, place, or grid. By using a frame, you can use those methods inside the frame, and then call create_window once for the frame. Drawing the text items directly on the canvas isn't very hard, so you might want to reconsider that approach if the frame-embedded-in-a-canvas solution seems too complex. Since you're creating a grid, the coordinates of each text item is going to be very easy to compute, especially if each row is the same height (which it probably is if you're using a single font). For drawing directly on the canvas, just figure out the line height of the font you're using (and there are commands for that). Then, each y coordinate is row*(lineheight+spacing). The x coordinate will be a fixed number based on the widest item in each column. If you give everything a tag for the column it is in, you can adjust the x coordinate and width of all items in a column with a single command. Object-oriented solution Here's an example of the frame-embedded-in-canvas solution, using an object-oriented approach: import tkinter as tk class Example(tk.Frame): def __init__(self, parent): tk.Frame.__init__(self, parent) self.canvas = tk.Canvas(self, borderwidth=0, background="#ffffff") self.frame = tk.Frame(self.canvas, background="#ffffff") self.vsb = tk.Scrollbar(self, orient="vertical", command=self.canvas.yview) self.canvas.configure(yscrollcommand=self.vsb.set) self.vsb.pack(side="right", fill="y") self.canvas.pack(side="left", fill="both", expand=True) self.canvas.create_window((4,4), window=self.frame, anchor="nw", tags="self.frame") self.frame.bind("<Configure>", self.onFrameConfigure) self.populate() def populate(self): '''Put in some fake data''' for row in range(100): tk.Label(self.frame, text="%s" % row, width=3, borderwidth="1", relief="solid").grid(row=row, column=0) t="this is the second column for row %s" %row tk.Label(self.frame, text=t).grid(row=row, column=1) def onFrameConfigure(self, event): '''Reset the scroll region to encompass the inner frame''' self.canvas.configure(scrollregion=self.canvas.bbox("all")) if __name__ == "__main__": root=tk.Tk() example = Example(root) example.pack(side="top", fill="both", expand=True) root.mainloop() Procedural solution Here is a solution that doesn't use a class: import tkinter as tk def populate(frame): '''Put in some fake data''' for row in range(100): tk.Label(frame, text="%s" % row, width=3, borderwidth="1", relief="solid").grid(row=row, column=0) t="this is the second column for row %s" %row tk.Label(frame, text=t).grid(row=row, column=1) def onFrameConfigure(canvas): '''Reset the scroll region to encompass the inner frame''' canvas.configure(scrollregion=canvas.bbox("all")) root = tk.Tk() canvas = tk.Canvas(root, borderwidth=0, background="#ffffff") frame = tk.Frame(canvas, background="#ffffff") vsb = tk.Scrollbar(root, orient="vertical", command=canvas.yview) canvas.configure(yscrollcommand=vsb.set) vsb.pack(side="right", fill="y") canvas.pack(side="left", fill="both", expand=True) canvas.create_window((4,4), window=frame, anchor="nw") frame.bind("<Configure>", lambda event, canvas=canvas: onFrameConfigure(canvas)) populate(frame) root.mainloop()


Instruction:
How to create a venv with a different Python version

I have different venvs in my machine in which I have Python 3.10. Now for a specific project, I realised that Python 3.10 is not suitable as some libraries are still not compatible. Therefore, when creating a new venv for a new project, I would like to downgrade Python, say to 3.8, only for this specific venv. How can I do that? What should I type onto the terminal to do this? PS: I use Visual Studio Code and its terminal to create venv.

Output: You can have multiple Python interpreter versions installed at the same time and you can create virtual environments with the needed version. Make sure you have installed the Python version you need and then specify its location when you create the virtual environment: virtualenv -p <path-to-new-python-installation> <new-venv-name> Example: virtualenv -p C:\Users\ssharma\AppData\Local\Programs\Python\Python38\python.exe venv38 This will create a virtual environment called venv38 with Python 3.8.


Instruction:
shutil.rmtree fails on Windows with 'Access is denied'

In Python, when running shutil.rmtree over a folder that contains a read-only file, the following exception is printed: File "C:\Python26\lib\shutil.py", line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python26\lib\shutil.py", line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python26\lib\shutil.py", line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python26\lib\shutil.py", line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python26\lib\shutil.py", line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python26\lib\shutil.py", line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python26\lib\shutil.py", line 216, in rmtree rmtree(fullname, ignore_errors, onerror) File "C:\Python26\lib\shutil.py", line 221, in rmtree onerror(os.remove, fullname, sys.exc_info()) File "C:\Python26\lib\shutil.py", line 219, in rmtree os.remove(fullname) WindowsError: [Error 5] Access is denied: 'build\\tcl\\tcl8.5\\msgs\\af.msg' Looking in File Properties dialog I noticed that af.msg file is set to be read-only. So the question is: what is the simplest workaround/fix to get around this problem - given that my intention is to do an equivalent of rm -rf build/ but on Windows? (without having to use third-party tools like unxutils or cygwin - as this code is targeted to be run on a bare Windows install with Python 2.6 w/ PyWin32 installed)

Output: Check this question out: What user do python scripts run as in windows? Apparently the answer is to change the file/folder to not be read-only and then remove it. Here's onerror() handler from pathutils.py mentioned by @Sridhar Ratnakumar in comments: def onerror(func, path, exc_info): """ Error handler for ``shutil.rmtree``. If the error is due to an access error (read only file) it attempts to add write permission and then retries. If the error is for another reason it re-raises the error. Usage : ``shutil.rmtree(path, onerror=onerror)`` """ import stat # Is the error an access error? if not os.access(path, os.W_OK): os.chmod(path, stat.S_IWUSR) func(path) else: raise


Instruction:
Error Updating Python3 pip AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'

I'm having an error when installing/updating any pip module in python3. Purging and reinstalling pip and every package I can thing of hasn't helped. Here's the error that I get in response to running python -m pip install --upgrade pip specifically (but the error is the same for attempting to install or update any pip module): Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/lib/python3/dist-packages/pip/__main__.py", line 16, in <module> from pip._internal.cli.main import main as _main # isort:skip # noqa File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 10, in <module> from pip._internal.cli.autocompletion import autocomplete File "/usr/lib/python3/dist-packages/pip/_internal/cli/autocompletion.py", line 9, in <module> from pip._internal.cli.main_parser import create_main_parser File "/usr/lib/python3/dist-packages/pip/_internal/cli/main_parser.py", line 7, in <module> from pip._internal.cli import cmdoptions File "/usr/lib/python3/dist-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module> from pip._internal.exceptions import CommandError File "/usr/lib/python3/dist-packages/pip/_internal/exceptions.py", line 10, in <module> from pip._vendor.six import iteritems File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 65, in <module> vendored("cachecontrol") File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 36, in vendored __import__(modulename, globals(), locals(), level=0) File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/adapter.py", line 5, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl/requests/__init__.py", line 95, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module> File "/home/patrick/.local/lib/python3.8/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/home/patrick/.local/lib/python3.8/site-packages/OpenSSL/crypto.py", line 3268, in <module> _lib.OpenSSL_add_all_algorithms() AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms' Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 72, in apport_excepthook from apport.fileutils import likely_packaged, get_recent_crashes File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module> from apport.report import Report File "/usr/lib/python3/dist-packages/apport/report.py", line 32, in <module> import apport.fileutils File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 12, in <module> import os, glob, subprocess, os.path, time, pwd, sys, requests_unixsocket File "/usr/lib/python3/dist-packages/requests_unixsocket/__init__.py", line 1, in <module> import requests File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl/requests/__init__.py", line 95, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module> File "/home/patrick/.local/lib/python3.8/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/home/patrick/.local/lib/python3.8/site-packages/OpenSSL/crypto.py", line 3268, in <module> _lib.OpenSSL_add_all_algorithms() AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms' Original exception was: Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/lib/python3/dist-packages/pip/__main__.py", line 16, in <module> from pip._internal.cli.main import main as _main # isort:skip # noqa File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 10, in <module> from pip._internal.cli.autocompletion import autocomplete File "/usr/lib/python3/dist-packages/pip/_internal/cli/autocompletion.py", line 9, in <module> from pip._internal.cli.main_parser import create_main_parser File "/usr/lib/python3/dist-packages/pip/_internal/cli/main_parser.py", line 7, in <module> from pip._internal.cli import cmdoptions File "/usr/lib/python3/dist-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module> from pip._internal.exceptions import CommandError File "/usr/lib/python3/dist-packages/pip/_internal/exceptions.py", line 10, in <module> from pip._vendor.six import iteritems File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 65, in <module> vendored("cachecontrol") File "/usr/lib/python3/dist-packages/pip/_vendor/__init__.py", line 36, in vendored __import__(modulename, globals(), locals(), level=0) File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/CacheControl-0.12.6-py2.py3-none-any.whl/cachecontrol/adapter.py", line 5, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl/requests/__init__.py", line 95, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/usr/share/python-wheels/urllib3-1.25.8-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module> File "/home/patrick/.local/lib/python3.8/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/home/patrick/.local/lib/python3.8/site-packages/OpenSSL/crypto.py", line 3268, in <module> _lib.OpenSSL_add_all_algorithms() AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms' I'm running Ubuntu 20.04 in WSL. Python openssl is already installed. sudo apt install python3-openssl Reading package lists... Done Building dependency tree Reading state information... Done python3-openssl is already the newest version (19.0.0-1build1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. My assumption is that I need to re-install some stuff, but I'm not sure what. I've tried the obvious stuff like python3-openssl, libssl-dev, libffi-dev, and python3-pip itself and python3 alltogether.

Output: As version 39.0.0 presented this bug, downgrading the cryptography package solves this, without purging or touching your OS. pip install cryptography==38.0.4 to downgrade from 39.0.0 which presented this error EDIT per @thomas, The error is a result of incompatibility between cryptography and pyopenssl, so if possible, also upgrading to openssl>22.1.0 should work: pip install -U pyopenssl cryptography


Instruction:
How to decorate all functions of a class without typing it over and over for each method?

Lets say my class has many methods, and I want to apply my decorator on each one of them, later when I add new methods, I want the same decorator to be applied, but I don't want to write @mydecorator above the method declaration all the time. If I look into __call__ is that the right way to go? I'd like to show this way, which is a similar solution to my problem for anybody finding this question later, using a mixin as mentioned in the comments. class WrapinMixin(object): def __call__(self, hey, you, *args): print 'entering', hey, you, repr(args) try: ret = getattr(self, hey)(you, *args) return ret except: ret = str(e) raise finally: print 'leaving', hey, repr(ret) Then you can in another class Wrapmymethodsaround(WrapinMixin): def __call__(self, hey, you, *args): return super(Wrapmymethodsaround, self).__call__(hey, you, *args) Editor's note: this example appears to be solving a different problem than what is asked about.

Output: Decorate the class with a function that walks through the class's attributes and decorates callables. This may be the wrong thing to do if you have class variables that may happen to be callable, and will also decorate nested classes (credits to Sven Marnach for pointing this out) but generally it's a rather clean and simple solution. Example implementation (note that this will not exclude special methods (__init__ etc.), which may or may not be desired): def for_all_methods(decorator): def decorate(cls): for attr in cls.__dict__: # there's propably a better way to do this if callable(getattr(cls, attr)): setattr(cls, attr, decorator(getattr(cls, attr))) return cls return decorate Use like this: @for_all_methods(mydecorator) class C(object): def m1(self): pass def m2(self, x): pass ...


Instruction:
Return in generator together with yield

In Python 2 it used to cause an error when return occurred together with yield inside a function definition. But for this code in Python 3.3: def f(): return 3 yield 2 x = f() print(x.__next__()) there is no error that return is used in function with yield. However when the function __next__ is called then there is thrown exception StopIteration. Why there is not just returned value 3? Is this return somehow ignored?

Output: This is a new feature in Python 3.3. Much like return in a generator has long been equivalent to raise StopIteration(), return <something> in a generator is now equivalent to raise StopIteration(<something>). For that reason, the exception you're seeing should be printed as StopIteration: 3, and the value is accessible through the attribute value on the exception object. If the generator is delegated to using the (also new) yield from syntax, it is the result. See PEP 380 for details. def f(): return 1 yield 2 def g(): x = yield from f() print(x) # g is still a generator so we need to iterate to run it: for _ in g(): pass This prints 1, but not 2.


Instruction:
What is this odd sorting algorithm?

Some answer originally had this sorting algorithm: for i from 0 to n-1: for j from 0 to n-1: if A[j] > A[i]: swap A[i] and A[j] Note that both i and j go the full range and thus j can be both larger and smaller than i, so it can make pairs both correct and wrong order (and it actually does do both!). I thought that's a mistake (and the author later called it that) and that this would jumble the array, but it does appear to sort correctly. It's not obvious why, though. But the code simplicity (going full ranges, and no +1 as in bubble sort) makes it interesting. Is it correct? If so, why does it work? And does it have a name? Python implementation with testing: from random import shuffle for _ in range(3): n = 20 A = list(range(n)) shuffle(A) print('before:', A) for i in range(n): for j in range(n): if A[j] > A[i]: A[i], A[j] = A[j], A[i] print('after: ', A, '\n') Sample output (Try it online!): before: [9, 14, 8, 12, 16, 19, 2, 1, 10, 11, 18, 4, 15, 3, 6, 17, 7, 0, 5, 13] after: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] before: [5, 1, 18, 10, 19, 14, 17, 7, 12, 16, 2, 0, 6, 8, 9, 11, 4, 3, 15, 13] after: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] before: [11, 15, 7, 14, 0, 2, 9, 4, 13, 17, 8, 10, 1, 12, 6, 16, 18, 3, 5, 19] after: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] Edit: Someone pointed out a very nice brand new paper about this algorithm. Just to clarify: We're unrelated, it's a coincidence. As far as I can tell it was submitted to arXiv before that answer that sparked my question and published by arXiv after my question.

Output: To prove that it's correct, you have to find some sort of invariant. Something that's true during every pass of the loop. Looking at it, after the very first pass of the inner loop, the largest element of the list will actually be in the first position. Now in the second pass of the inner loop, i = 1, and the very first comparison is between i = 1 and j = 0. So, the largest element was in position 0, and after this comparison, it will be swapped to position 1. In general, then it's not hard to see that after each step of the outer loop, the largest element will have moved one to the right. So after the full steps, we know at least the largest element will be in the correct position. What about all the rest? Let's say the second-largest element sits at position i of the current loop. We know that the largest element sits at position i-1 as per the previous discussion. Counter j starts at 0. So now we're looking for the first A[j] such that it's A[j] > A[i]. Well, the A[i] is the second largest element, so the first time that happens is when j = i-1, at the first largest element. Thus, they're adjacent and get swapped, and are now in the "right" order. Now A[i] again points to the largest element, and hence for the rest of the inner loop no more swaps are performed. So we can say: Once the outer loop index has moved past the location of the second largest element, the second and first largest elements will be in the right order. They will now slide up together, in every iteration of the outer loop, so we know that at the end of the algorithm both the first and second-largest elements will be in the right position. What about the third-largest element? Well, we can use the same logic again: Once the outer loop counter i is at the position of the third-largest element, it'll be swapped such that it'll be just below the second largest element (if we have found that one already!) or otherwise just below the first largest element. Ah. And here we now have our invariant: After k iterations of the outer loop, the k-length sequence of elements, ending at position k-1, will be in sorted order: After the 1st iteration, the 1-length sequence, at position 0, will be in the correct order. That's trivial. After the 2nd iteration, we know the largest element is at position 1, so obviously the sequence A[0], A[1] is in the correct order. Now let's assume we're at step k, so all the elements up to position k-1 will be in order. Now i = k and we iterate over j. What this does is basically find the position at which the new element needs to be slotted into the existing sorted sequence so that it'll be properly sorted. Once that happens, the rest of the elements "bubble one up" until now the largest element sits at position i = k and no further swaps happen. Thus finally at the end of step N, all the elements up to position N-1 are in the correct order, QED.


Instruction:
Python auto-import extension for VSCode

Is there a Python auto-import extension/plugin available for VSCode? By auto-import I mean automatically importing of Python modules, so if you type sys.argv then it should automatically import the sys module. Eclipse and IntelliJ have this feature with Java.

Output: VSCode team recently released Pylance Features Docstrings Signature help, with type information Parameter suggestions Code completion Auto-imports (as well as add and remove import code actions) As-you-type reporting of code errors and warnings (diagnostics) Code outline Code navigation Code lens (references/implementations) Type checking mode Native multi-root workspace support IntelliCode compatibility Jupyter Notebooks compatibility


Instruction:
Should enum instances be compared by identity or equality?

PEP 8 Programming Recommendations says: Comparisons to singletons like None should always be done with is or is not, never the equality operators. According to the docs, enum members are singletons. Does that mean they should also be compared by identity? class Color(Enum): RED = 1 GREEN = 2 BLUE = 3 # like this? if color is Color.RED: ... # or like this if color == Color.RED: ... When using equality operators, I haven't noticed any issues with this to warrant such strong wording as PEP 8. What's the drawback of using equality, if any? Doesn't it just fall back to an identity-based comparison anyway? Is this just a micro-optimisation?

Output: From https://docs.python.org/3/howto/enum.html#comparisons : Enumeration members are compared by identity: >>> Color.RED is Color.RED True >>> Color.RED is Color.BLUE False >>> Color.RED is not Color.BLUE True


Instruction:
How to extract a filename from a URL and append a word to it?

I have the following URL: url = http://photographs.500px.com/kyle/09-09-201315-47-571378756077.jpg I would like to extract the file name in this URL: 09-09-201315-47-571378756077.jpg Once I get this file name, I'm going to save it with this name to the Desktop. filename = **extracted file name from the url** download_photo = urllib.urlretrieve(url, "/home/ubuntu/Desktop/%s.jpg" % (filename)) After this, I'm going to resize the photo, once that is done, I've going to save the resized version and append the word "_small" to the end of the filename. downloadedphoto = Image.open("/home/ubuntu/Desktop/%s.jpg" % (filename)) resize_downloadedphoto = downloadedphoto.resize.((300, 300), Image.ANTIALIAS) resize_downloadedphoto.save("/home/ubuntu/Desktop/%s.jpg" % (filename + _small)) From this, what I am trying to achieve is to get two files, the original photo with the original name, then the resized photo with the modified name. Like so: 09-09-201315-47-571378756077.jpg rename to: 09-09-201315-47-571378756077_small.jpg How can I go about doing this?

Output: You can use urllib.parse.urlparse with os.path.basename: import os from urllib.parse import urlparse url = "http://photographs.500px.com/kyle/09-09-201315-47-571378756077.jpg" a = urlparse(url) print(a.path) # Output: /kyle/09-09-201315-47-571378756077.jpg print(os.path.basename(a.path)) # Output: 09-09-201315-47-571378756077.jpg Your URL might contain percent-encoded characters like %20 for space or %E7%89%B9%E8%89%B2 for "特色". If that's the case, you'll need to unquote (or unquote_plus) them. You can also use pathlib.Path().name instead of os.path.basename, which could help to add a suffix in the name (like asked in the original question): from pathlib import Path, PurePosixPath from urllib.parse import urlparse, unquote url = "http://photographs.500px.com/kyle/09-09-2013%20-%2015-47-571378756077.jpg" urlparse(url).path url_parsed = urlparse(url) print(unquote(url_parsed.path)) # Output: /kyle/09-09-2013 - 15-47-571378756077.jpg file_path = Path("/home/ubuntu/Desktop/") / unquote(PurePosixPath(url_parsed.path).name) print(file_path) # Output: /home/ubuntu/Desktop/09-09-2013 - 15-47-571378756077.jpg new_file = file_path.with_stem(file_path.stem + "_small") print(new_file) # Output: /home/ubuntu/Desktop/09-09-2013 - 15-47-571378756077_small.jpg For parsing URL paths, note that pathlib.PurePosixPath is more specific and more applicable than pathlib.Path. Also, a less-robust alternative is to use unquote(urlparse(url).path.split("/")[-1]).


Instruction:
Ordinal numbers replacement

I am currently looking for the way to replace words like first, second, third,...with appropriate ordinal number representation (1st, 2nd, 3rd). I have been googling for the last week and I didn't find any useful standard tool or any function from NLTK. So is there any or should I write some regular expressions manually? Thanks for any advice

Output: The package number-parser can parse ordinal words ("first", "second", etc) to integers. from number_parser import parse_ordinal n = parse_ordinal("first") To convert an integer to "1st", "2nd", etc, you can use the following: def ordinal(n: int): if 11 <= (n % 100) <= 13: suffix = 'th' else: suffix = ['th', 'st', 'nd', 'rd', 'th'][min(n % 10, 4)] return str(n) + suffix Here is a more terse but less readable version (taken from Gareth on codegolf): ordinal = lambda n: "%d%s" % (n,"tsnrhtdd"[(n//10%10!=1)*(n%10<4)*n%10::4]) This works on any number: print([ordinal(n) for n in range(1,32)]) ['1st', '2nd', '3rd', '4th', '5th', '6th', '7th', '8th', '9th', '10th', '11th', '12th', '13th', '14th', '15th', '16th', '17th', '18th', '19th', '20th', '21st', '22nd', '23rd', '24th', '25th', '26th', '27th', '28th', '29th', '30th', '31st']