Instruction:
">", "<", ">=" and "<=" don't work with "filter()" in Django

With = below, I could filter persons by age: qs = Person.objects.filter(age = 20) # ↑ Here But with >, <, >= and <= below, I couldn't filter persons by age: qs = Person.objects.filter(age > 20) # ↑ Here qs = Person.objects.filter(age < 20) # ↑ Here qs = Person.objects.filter(age >= 20) # ↑↑ Here qs = Person.objects.filter(age <= 20) # ↑↑ Here Then, I got the error below: NameError: name 'age' is not defined How can I do greater than(>), greater than or equal to(>=), less than(<) and less than or equal to(>=) with filter() in Django?

Output: Greater than: Person.objects.filter(age__gt=20) Greater than or equal to: Person.objects.filter(age__gte=20) Less than: Person.objects.filter(age__lt=20) Less than or equal to: Person.objects.filter(age__lte=20) You can find them all in [the documentation].(https://docs.djangoproject.com/en/stable/ref/models/querysets/).


Instruction:
How does tuple comparison work in Python?

I have been reading the Core Python programming book, and the author shows an example like: (4, 5) < (3, 5) # Equals false So, I'm wondering, how/why does it equal false? How does python compare these two tuples? Btw, it's not explained in the book.

Output: Tuples are compared position by position: the first item of the first tuple is compared to the first item of the second tuple; if they are not equal (i.e. the first is greater or smaller than the second) then that's the result of the comparison, else the second item is considered, then the third and so on. See Common Sequence Operations: Sequences of the same type also support comparisons. In particular, tuples and lists are compared lexicographically by comparing corresponding elements. This means that to compare equal, every element must compare equal and the two sequences must be of the same type and have the same length. Also Value Comparisons for further details: Lexicographical comparison between built-in collections works as follows: For two collections to compare equal, they must be of the same type, have the same length, and each pair of corresponding elements must compare equal (for example, [1,2] == (1,2) is false because the type is not the same). Collections that support order comparison are ordered the same as their first unequal elements (for example, [1,2,x] <= [1,2,y] has the same value as x <= y). If a corresponding element does not exist, the shorter collection is ordered first (for example, [1,2] < [1,2,3] is true). If not equal, the sequences are ordered the same as their first differing elements. For example, cmp([1,2,x], [1,2,y]) returns the same as cmp(x,y). If the corresponding element does not exist, the shorter sequence is considered smaller (for example, [1,2] < [1,2,3] returns True). Note 1: < and > do not mean "smaller than" and "greater than" but "is before" and "is after": so (0, 1) "is before" (1, 0). Note 2: tuples must not be considered as vectors in a n-dimensional space, compared according to their length. Note 3: referring to question https://stackoverflow.com/questions/36911617/python-2-tuple-comparison: do not think that a tuple is "greater" than another only if any element of the first is greater than the corresponding one in the second. Note 4: as @david Winiecki mentioned in the comments, in case of two tuples of different length, the first one which reaches its end, being the previous part equal, is declared as the lower: (1, 2) < (1, 2, 3), since 1=1, 2=2 and then the first tuple ends


Instruction:
How can I run Conda?

I installed Anaconda and can run Python, so I assume that I installed it correctly. Following this introductory documentation, I am trying to install Python v3.3, so I am copying and pasting the following line into my console: conda create -n py33 python=3.3 anaconda However, that gives me an error: -bash: conda: command not found What do I need to do to run Conda? I am working on a Linux system.

Output: It turns out that I had not set the path. To do so, I first had to edit .bash_profile (I downloaded it to my local desktop to do that; I do not know how to text edit a file from Linux) Then add this to .bash_profile: PATH=$PATH:$HOME/anaconda/bin


Instruction:
From conda create requirements.txt for pip3

I usually use conda to manage my environments, but now I am on a project that needs a little more horsepower than my laptop. So I am trying to use my university's workstations which have new Intel Xeons. But I don't have admin rights and the workstation does not have conda so I am forced to work with virtualenv and pip3. How do I generate a requirements.txt from conda that will work with pip3 and venv? conda list -e > requirements.txt does not generate a compatible file: = is not a valid operator. Did you mean == ? The conda output is: # This file may be used to create an environment using: # $ conda create --name <env> --file <this file> # platform: osx-64 certifi=2016.2.28=py36_0 cycler=0.10.0=py36_0 freetype=2.5.5=2 icu=54.1=0 libpng=1.6.30=1 matplotlib=2.0.2=np113py36_0 mkl=2017.0.3=0 numpy=1.13.1=py36_0 openssl=1.0.2l=0 pip=9.0.1=py36_1 pyparsing=2.2.0=py36_0 pyqt=5.6.0=py36_2 python=3.6.2=0 python-dateutil=2.6.1=py36_0 pytz=2017.2=py36_0 qt=5.6.2=2 readline=6.2=2 scikit-learn=0.19.0=np113py36_0 scipy=0.19.1=np113py36_0 setuptools=36.4.0=py36_1 sip=4.18=py36_0 six=1.10.0=py36_0 sqlite=3.13.0=0 tk=8.5.18=0 wheel=0.29.0=py36_0 xz=5.2.3=0 zlib=1.2.11=0 I thought I would just manually change all = to == but the there are two = in the conda output. Which one to change? Surely there is an easier way? EDIT: pip freeze > requirements.txt gives: certifi==2016.2.28 cycler==0.10.0 matplotlib==2.0.2 matplotlib-venn==0.11.5 numpy==1.13.1 pyparsing==2.2.0 python-dateutil==2.6.1 pytz==2017.2 scikit-learn==0.19.0 scipy==0.19.1 six==1.10.0

Output: As the comment at the top indicates, the output of conda list -e > requirements.txt can be used to create a conda virtual environment with conda create --name <env> --file requirements.txt but this output isn't in the right format for pip. If you want a file which you can use to create a pip virtual environment (i.e. a requirements.txt in the right format) you can install pip within the conda environment, then use pip to create requirements.txt. conda activate <env> conda install pip pip freeze > requirements.txt Then use the resulting requirements.txt to create a pip virtual environment: python3 -m venv env source env/bin/activate pip install -r requirements.txt When I tested this, the packages weren't identical across the outputs (pip included fewer packages) but it was sufficient to set up a functional environment. For those getting odd path references in requirements.txt, use: pip list --format=freeze > requirements.txt


Instruction:
Split a string by a delimiter in Python

Consider the following input string: 'MATCHES__STRING' I want to split that string wherever the "delimiter" __ occurs. This should output a list of strings: ['MATCHES', 'STRING'] To split on whitespace, see How do I split a string into a list of words?. To extract everything before the first delimiter, see Splitting on first occurrence. To extract everything before the last delimiter, see Partition string in Python and get value of last segment after colon.

Output: Use the str.split method: >>> "MATCHES__STRING".split("__") ['MATCHES', 'STRING']


Instruction:
Round to 5 (or other number) in Python

Is there a built-in function that can round like the following? 10 -> 10 12 -> 10 13 -> 15 14 -> 15 16 -> 15 18 -> 20

Output: I don't know of a standard function in Python, but this works for me: Python 3 def myround(x, base=5): return base * round(x/base) It is easy to see why the above works. You want to make sure that your number divided by 5 is an integer, correctly rounded. So, we first do exactly that (round(x/5)), and then since we divided by 5, we multiply by 5 as well. I made the function more generic by giving it a base parameter, defaulting to 5. Python 2 In Python 2, float(x) would be needed to ensure that / does floating-point division, and a final conversion to int is needed because round() returns a floating-point value in Python 2. def myround(x, base=5): return int(base * round(float(x)/base))


Instruction:
Could not install packages due to an EnvironmentError: [Errno 13]

In my MacOS Mojave terminal I wanted to install a python package with pip. At the end it says: You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. So I wanted to update pip with the given command but I got an error: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/pip-18.0-py2.7.egg/EGG-INFO/PKG-INFO' Consider using the `--user` option or check the permissions. I don't really understand what to do now. Also I realized it says Python 2.7 in the error message but I have and want to use only python 3.

Output: If you want to use python3+ to install the packages you need to use pip3 install package_name And to solve the errno 13 you have to add --user at the end pip3 install package_name --user EDIT: For any project in Python it's highly recommended to work on a Virtual environment, which is a tool that helps to keep dependencies required by different projects separate by creating isolated Python virtual environments for them. In order to create one with python3+ you have to use the following command: virtualenv enviroment_name -p python3 And then you work on it just by activating it: source enviroment_name/bin/activate Once the virtual environment is activated, the name of your virtual environment will appear on the left side of the terminal. This will let you know that the virtual environment is currently active. Now you can install dependencies related to the project in this virtual environment by just using pip. pip install package_name


Instruction:
How can I choose a custom string representation for a class itself (not instances of the class)?

Consider this class: class foo(object): pass The default string representation looks something like this: >>> str(foo) "<class '__main__.foo'>" How can I make this display a custom string? See How to print instances of a class using print()? for the corresponding question about instances of the class. In fact, this question is really a special case of that one - because in Python, classes are themselves also objects belonging to their own class - but it's not directly obvious how to apply the advice, since the default "class of classes" is pre-defined.

Output: Implement __str__() or __repr__() in the class's metaclass. class MC(type): def __repr__(self): return 'Wahaha!' class C(object): __metaclass__ = MC print(C) Use __str__ if you mean a readable stringification, use __repr__ for unambiguous representations. Edit: Python 3 Version class MC(type): def __repr__(self): return 'Wahaha!' class C(object, metaclass=MC): pass print(C)


Instruction:
What is the result of % (modulo operator / percent sign) in Python?

What does the % do in a calculation? I can't seem to work out what it does. Does it work out a percent of the calculation for example: 4 % 2 is apparently equal to 0. How?

Output: The % (modulo) operator yields the remainder from the division of the first argument by the second. The numeric arguments are first converted to a common type. A zero right argument raises the ZeroDivisionError exception. The arguments may be floating point numbers, e.g., 3.14%0.7 equals 0.34 (since 3.14 equals 4*0.7 + 0.34.) The modulo operator always yields a result with the same sign as its second operand (or zero); the absolute value of the result is strictly smaller than the absolute value of the second operand [2]. Taken from https://docs.python.org/reference/expressions.html#binary-arithmetic-operations Example 1: 6%2 evaluates to 0 because there's no remainder if 6 is divided by 2 (3 times). Example 2: 7%2 evaluates to 1 because there's a remainder of 1 when 7 is divided by 2 (3 times). So to summarise that, it returns the remainder of a division operation, or 0 if there is no remainder. So 6%2 means find the remainder of 6 divided by 2.


Instruction:
Configuring so that pip install can work from github

We'd like to use pip with github to install private packages to our production servers. This question concerns what needs to be in the github repo in order for the install to be successful. Assuming the following command line (which authenticates just fine and tries to install): pip install git+ssh://git@github.com/BlahCo/search/tree/prod_release_branch/ProductName What needs to reside in the ProductName? Is it the contents of what would normally be in the tar file after running setup.py with the sdist option, or is the actual tar.gz file, or something else? I'm asking here because I've tried several variations and can't make it work. Any help appreciated.

Output: You need the whole python package, with a setup.py file in it. A package named foo would be: foo # the installable package ├── foo │ ├── __init__.py │ └── bar.py └── setup.py And install from github like: $ pip install git+ssh://git@github.com/myuser/foo.git or $ pip install git+https://github.com/myuser/foo.git@v123 or $ pip install git+https://github.com/myuser/foo.git@newbranch More info at https://pip.pypa.io/en/stable/cli/pip_install/


Instruction:
Importing module from string variable using "__import__" gives different results than a normal import statement

I'm working on a documentation (personal) for nested matplotlib (MPL) library, which differs from MPL own provided, by interested submodule packages. I'm writing Python script which I hope will automate document generation from future MPL releases. I selected interested submodules/packages and want to list their main classes from which I'll generate list and process it with pydoc. The problem is that I can't find a way to instruct Python to load a submodule from a string. Here is an example of what I tried: import matplotlib.text as text x = dir(text) i = __import__('matplotlib.text') y = dir(i) j = __import__('matplotlib') z = dir(j) And here is a 3-way comparison of above lists through pprint: I don't understand what's loaded in y object - it's base matplotlib plus something else, but it lacks information that I wanted and that is main classes from matplotlib.text package. It's the top blue coloured part on screenshot (x list).

Output: The __import__ function can be a bit hard to understand. If you change i = __import__('matplotlib.text') to i = __import__('matplotlib.text', fromlist=['']) then i will refer to matplotlib.text. In Python 3.1 or later, you can use importlib: import importlib i = importlib.import_module("matplotlib.text") Some notes If you're trying to import something from a sub-folder e.g. ./feature/email.py, the code will look like importlib.import_module("feature.email") Before Python 3.3 you could not import anything if there was no __init__.py in the folder with file you were trying to import (see caveats before deciding if you want to keep the file for backward compatibility e.g. with pytest).


Instruction:
How can I use newline '\n' in an f-string to format a list of strings?

I tried this code: names = ['Adam', 'Bob', 'Cyril'] text = f"Winners are:\n{'\n'.join(names)}" print(text) However, '\' cannot be used inside the {...} expression portions of an f-string. How can I make it work? The result should be: Winners are: Adam Bob Cyril See Why isn't it possible to use backslashes inside the braces of f-strings? How can I work around the problem? for some additional discussion of why the limitation exists.

Output: Python 3.12+ You can use backslashes within f-strings and the existing code from the question works as expected. See https://docs.python.org/3.12/whatsnew/3.12.html#pep-701-syntactic-formalization-of-f-strings. Python < 3.12 You can't. Backslashes cannot appear inside the curly braces {}; doing so results in a SyntaxError: >>> f'{\}' SyntaxError: f-string expression part cannot include a backslash This is specified in the PEP for f-strings: Backslashes may not appear inside the expression portions of f-strings, [...] One option is assigning '\n' to a name and then .join on that inside the f-string; that is, without using a literal: names = ['Adam', 'Bob', 'Cyril'] nl = '\n' text = f"Winners are:{nl}{nl.join(names)}" print(text) Results in: Winners are: Adam Bob Cyril Another option, as specified by @wim, is to use chr(10) to get \n returned and then join there. f"Winners are:\n{chr(10).join(names)}" Yet another, of course, is to '\n'.join beforehand and then add the name accordingly: n = "\n".join(names) text = f"Winners are:\n{n}" which results in the same output. Note: This is one of the small differences between f-strings and str.format. In the latter, you can always use punctuation granted that a corresponding wacky dict is unpacked that contains those keys: >>> "{\\} {*}".format(**{"\\": 'Hello', "*": 'World!'}) "Hello World!" (Please don't do this.) In the former, punctuation isn't allowed because you can't have identifiers that use them. Aside: I would definitely opt for print or format, as the other answers suggest as an alternative. The options I've given only apply if you must for some reason use f-strings. Just because something is new, doesn't mean you should try and do everything with it ;-)


Instruction:
How to set xlim and ylim for a subplot

I would like to limit the X and Y axis in matplotlib for a specific subplot. The subplot figure itself doesn't have any axis property. I want for example to change only the limits for the second plot: import matplotlib.pyplot as plt fig=plt.subplot(131) plt.scatter([1,2],[3,4]) fig=plt.subplot(132) plt.scatter([10,20],[30,40]) fig=plt.subplot(133) plt.scatter([15,23],[35,43]) plt.show()

Output: You should use the object-oriented interface to matplotlib, rather than the state machine interface. Almost all of the plt.* function are thin wrappers that basically do gca().*. plt.subplot returns an axes object. Once you have a reference to the axes object you can plot directly to it, change its limits, etc. import matplotlib.pyplot as plt ax1 = plt.subplot(131) ax1.scatter([1, 2], [3, 4]) ax1.set_xlim([0, 5]) ax1.set_ylim([0, 5]) ax2 = plt.subplot(132) ax2.scatter([1, 2],[3, 4]) ax2.set_xlim([0, 5]) ax2.set_ylim([0, 5]) and so on for as many axes as you want. Or better, wrap it all up in a loop: import matplotlib.pyplot as plt DATA_x = ([1, 2], [2, 3], [3, 4]) DATA_y = DATA_x[::-1] XLIMS = [[0, 10]] * 3 YLIMS = [[0, 10]] * 3 for j, (x, y, xlim, ylim) in enumerate(zip(DATA_x, DATA_y, XLIMS, YLIMS)): ax = plt.subplot(1, 3, j + 1) ax.scatter(x, y) ax.set_xlim(xlim) ax.set_ylim(ylim)


Instruction:
How to count the number of true elements in a NumPy bool array

I have a NumPy array 'boolarr' of boolean type. I want to count the number of elements whose values are True. Is there a NumPy or Python routine dedicated for this task? Or, do I need to iterate over the elements in my script?

Output: You have multiple options. Two options are the following. boolarr.sum() numpy.count_nonzero(boolarr) Here's an example: >>> import numpy as np >>> boolarr = np.array([[0, 0, 1], [1, 0, 1], [1, 0, 1]], dtype=np.bool) >>> boolarr array([[False, False, True], [ True, False, True], [ True, False, True]], dtype=bool) >>> boolarr.sum() 5 Of course, that is a bool-specific answer. More generally, you can use numpy.count_nonzero. >>> np.count_nonzero(boolarr) 5


Instruction:
How to split a dos path into its components in Python

I have a string variable which represents a dos path e.g: var = "d:\stuff\morestuff\furtherdown\THEFILE.txt" I want to split this string into: [ "d", "stuff", "morestuff", "furtherdown", "THEFILE.txt" ] I have tried using split() and replace() but they either only process the first backslash or they insert hex numbers into the string. I need to convert this string variable into a raw string somehow so that I can parse it. What's the best way to do this? I should also add that the contents of var i.e. the path that I'm trying to parse, is actually the return value of a command line query. It's not path data that I generate myself. Its stored in a file, and the command line tool is not going to escape the backslashes.

Output: I've been bitten loads of times by people writing their own path fiddling functions and getting it wrong. Spaces, slashes, backslashes, colons -- the possibilities for confusion are not endless, but mistakes are easily made anyway. So I'm a stickler for the use of os.path, and recommend it on that basis. (However, the path to virtue is not the one most easily taken, and many people when finding this are tempted to take a slippery path straight to damnation. They won't realise until one day everything falls to pieces, and they -- or, more likely, somebody else -- has to work out why everything has gone wrong, and it turns out somebody made a filename that mixes slashes and backslashes -- and some person suggests that the answer is "not to do that". Don't be any of these people. Except for the one who mixed up slashes and backslashes -- you could be them if you like.) You can get the drive and path+file like this: drive, path_and_file = os.path.splitdrive(path) Get the path and the file: path, file = os.path.split(path_and_file) Getting the individual folder names is not especially convenient, but it is the sort of honest middling discomfort that heightens the pleasure of later finding something that actually works well: folders = [] while 1: path, folder = os.path.split(path) if folder != "": folders.append(folder) else: if path != "": folders.append(path) break folders.reverse() (This pops a "\" at the start of folders if the path was originally absolute. You could lose a bit of code if you didn't want that.)


Instruction:
Install a specific (ana)conda package version

I want to install the 'rope' package in my current active environment using conda. Currently, the following 'rope' versions are available: (data_downloader) ~/code/data_downloader $ conda search rope Using Anaconda Cloud api site https://api.anaconda.org Fetching package metadata: .... cached-property 1.2.0 py27_0 defaults 1.2.0 py34_0 defaults 1.2.0 py35_0 defaults 1.3.0 py27_0 defaults 1.3.0 py34_0 defaults 1.3.0 py35_0 defaults rope 0.9.4 py26_0 defaults 0.9.4 py27_0 defaults 0.9.4 py33_0 defaults 0.9.4 py34_0 defaults 0.9.4 py26_1 defaults 0.9.4 py27_1 defaults 0.9.4 py33_1 defaults 0.9.4 py34_1 defaults . 0.9.4 py35_1 defaults I would like to install 1.3.0 py35_0 defaults I've tried all sorts of permutations of conda install which I'm not going to list here because none of them are correct. I am also not sure what the py35_0 is (I'm assuming this is the version of the python against which the package was built?) and I also don't know what 'defaults' means?

Output: There is no version 1.3.0 for rope. 1.3.0 refers to the package cached-property. The highest available version of rope is 0.9.4. You can install different versions with conda install package=version. But in this case there is only one version of rope so you don't need that. The reason you see the cached-property in this listing is because it contains the string "rope": "cached-p rope erty" py35_0 means that you need python version 3.5 for this specific version. If you only have python3.4 and the package is only for version 3.5 you cannot install it with conda. I am not quite sure on the defaults either. It should be an indication that this package is inside the default conda channel.


Instruction:
Django model "doesn't declare an explicit app_label"

After a dozen hours of troubleshooting, I thought I was finally in business, but then I got: Model class django.contrib.contenttypes.models.ContentType doesn't declare an explicit app_label There is so little info on this on the web, and no solution out there has resolved my issue. I'm using Python 3.4 and Django 1.10. From my settings.py: INSTALLED_APPS = [ 'DeleteNote.apps.DeletenoteConfig', 'LibrarySync.apps.LibrarysyncConfig', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', ] And my apps.py files look like this: from django.apps import AppConfig class DeletenoteConfig(AppConfig): name = 'DeleteNote' and from django.apps import AppConfig class LibrarysyncConfig(AppConfig): name = 'LibrarySync'

Output: Are you missing putting in your application name into the settings file? The myAppNameConfig is the default class generated at apps.py by the .manage.py createapp myAppName command. Where myAppName is the name of your app. settings.py INSTALLED_APPS = [ 'myAppName.apps.myAppNameConfig', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', ] This way, the settings file finds out what you want to call your application. You can change how it looks later in the apps.py file by adding the following code in myAppName/apps.py class myAppNameConfig(AppConfig): name = 'myAppName' verbose_name = 'A Much Better Name'


Instruction:
How to replace text in a string column of a Pandas dataframe?

I have a column in my dataframe like this: range "(2,30)" "(50,290)" "(400,1000)" ... and I want to replace the , comma with - dash. I'm currently using this method but nothing is changed. org_info_exc['range'].replace(',', '-', inplace=True) Can anybody help?

Output: Use the vectorised str method replace: df['range'] = df['range'].str.replace(',','-') df range 0 (2-30) 1 (50-290) EDIT: so if we look at what you tried and why it didn't work: df['range'].replace(',','-',inplace=True) from the docs we see this description: str or regex: str: string exactly matching to_replace will be replaced with value So because the str values do not match, no replacement occurs, compare with the following: df = pd.DataFrame({'range':['(2,30)',',']}) df['range'].replace(',','-', inplace=True) df['range'] 0 (2,30) 1 - Name: range, dtype: object here we get an exact match on the second row and the replacement occurs.


Instruction:
Pandas column of lists, create a row for each list element

I have a dataframe where some cells contain lists of multiple values. Rather than storing multiple values in a cell, I'd like to expand the dataframe so that each item in the list gets its own row (with the same values in all other columns). So if I have: import pandas as pd import numpy as np df = pd.DataFrame( {'trial_num': [1, 2, 3, 1, 2, 3], 'subject': [1, 1, 1, 2, 2, 2], 'samples': [list(np.random.randn(3).round(2)) for i in range(6)] } ) df Out[10]: samples subject trial_num 0 [0.57, -0.83, 1.44] 1 1 1 [-0.01, 1.13, 0.36] 1 2 2 [1.18, -1.46, -0.94] 1 3 3 [-0.08, -4.22, -2.05] 2 1 4 [0.72, 0.79, 0.53] 2 2 5 [0.4, -0.32, -0.13] 2 3 How do I convert to long form, e.g.: subject trial_num sample sample_num 0 1 1 0.57 0 1 1 1 -0.83 1 2 1 1 1.44 2 3 1 2 -0.01 0 4 1 2 1.13 1 5 1 2 0.36 2 6 1 3 1.18 0 # etc. The index is not important, it's OK to set existing columns as the index and the final ordering isn't important.

Output: UPDATE: the solution below was helpful for older Pandas versions, because the DataFrame.explode() wasn’t available. Starting from Pandas 0.25.0 you can simply use DataFrame.explode(). lst_col = 'samples' r = pd.DataFrame({ col:np.repeat(df[col].values, df[lst_col].str.len()) for col in df.columns.drop(lst_col)} ).assign(**{lst_col:np.concatenate(df[lst_col].values)})[df.columns] Result: In [103]: r Out[103]: samples subject trial_num 0 0.10 1 1 1 -0.20 1 1 2 0.05 1 1 3 0.25 1 2 4 1.32 1 2 5 -0.17 1 2 6 0.64 1 3 7 -0.22 1 3 8 -0.71 1 3 9 -0.03 2 1 10 -0.65 2 1 11 0.76 2 1 12 1.77 2 2 13 0.89 2 2 14 0.65 2 2 15 -0.98 2 3 16 0.65 2 3 17 -0.30 2 3 PS here you may find a bit more generic solution UPDATE: some explanations: IMO the easiest way to understand this code is to try to execute it step-by-step: in the following line we are repeating values in one column N times where N - is the length of the corresponding list: In [10]: np.repeat(df['trial_num'].values, df[lst_col].str.len()) Out[10]: array([1, 1, 1, 2, 2, 2, 3, 3, 3, 1, 1, 1, 2, 2, 2, 3, 3, 3], dtype=int64) this can be generalized for all columns, containing scalar values: In [11]: pd.DataFrame({ ...: col:np.repeat(df[col].values, df[lst_col].str.len()) ...: for col in df.columns.drop(lst_col)} ...: ) Out[11]: trial_num subject 0 1 1 1 1 1 2 1 1 3 2 1 4 2 1 5 2 1 6 3 1 .. ... ... 11 1 2 12 2 2 13 2 2 14 2 2 15 3 2 16 3 2 17 3 2 [18 rows x 2 columns] using np.concatenate() we can flatten all values in the list column (samples) and get a 1D vector: In [12]: np.concatenate(df[lst_col].values) Out[12]: array([-1.04, -0.58, -1.32, 0.82, -0.59, -0.34, 0.25, 2.09, 0.12, 0.83, -0.88, 0.68, 0.55, -0.56, 0.65, -0.04, 0.36, -0.31]) putting all this together: In [13]: pd.DataFrame({ ...: col:np.repeat(df[col].values, df[lst_col].str.len()) ...: for col in df.columns.drop(lst_col)} ...: ).assign(**{lst_col:np.concatenate(df[lst_col].values)}) Out[13]: trial_num subject samples 0 1 1 -1.04 1 1 1 -0.58 2 1 1 -1.32 3 2 1 0.82 4 2 1 -0.59 5 2 1 -0.34 6 3 1 0.25 .. ... ... ... 11 1 2 0.68 12 2 2 0.55 13 2 2 -0.56 14 2 2 0.65 15 3 2 -0.04 16 3 2 0.36 17 3 2 -0.31 [18 rows x 3 columns] using pd.DataFrame()[df.columns] will guarantee that we are selecting columns in the original order...


Instruction:
Why can a function modify some arguments as perceived by the caller, but not others?

I'm trying to understand Python's approach to variable scope. In this example, why is f() able to alter the value of x, as perceived within main(), but not the value of n? def f(n, x): n = 2 x.append(4) print('In f():', n, x) def main(): n = 1 x = [0,1,2,3] print('Before:', n, x) f(n, x) print('After: ', n, x) main() Output: Before: 1 [0, 1, 2, 3] In f(): 2 [0, 1, 2, 3, 4] After: 1 [0, 1, 2, 3, 4] See also: How do I pass a variable by reference? Are Python variables pointers? Or else, what are they?

Output: Some answers contain the word "copy" in the context of a function call. I find it confusing. Python doesn't copy objects you pass during a function call ever. Function parameters are names. When you call a function, Python binds these parameters to whatever objects you pass (via names in a caller scope). Objects can be mutable (like lists) or immutable (like integers and strings in Python). A mutable object you can change. You can't change a name, you just can bind it to another object. Your example is not about scopes or namespaces, it is about naming and binding and mutability of an object in Python. def f(n, x): # these `n`, `x` have nothing to do with `n` and `x` from main() n = 2 # put `n` label on `2` balloon x.append(4) # call `append` method of whatever object `x` is referring to. print('In f():', n, x) x = [] # put `x` label on `[]` ballon # x = [] has no effect on the original list that is passed into the function Here are nice pictures on the difference between variables in other languages and names in Python.


Instruction:
Save classifier to disk in scikit-learn

How do I save a trained Naive Bayes classifier to disk and use it to predict data? I have the following sample program from the scikit-learn website: from sklearn import datasets iris = datasets.load_iris() from sklearn.naive_bayes import GaussianNB gnb = GaussianNB() y_pred = gnb.fit(iris.data, iris.target).predict(iris.data) print "Number of mislabeled points : %d" % (iris.target != y_pred).sum()

Output: Classifiers are just objects that can be pickled and dumped like any other. To continue your example: import cPickle # save the classifier with open('my_dumped_classifier.pkl', 'wb') as fid: cPickle.dump(gnb, fid) # load it again with open('my_dumped_classifier.pkl', 'rb') as fid: gnb_loaded = cPickle.load(fid)


Instruction:
Python list directory, subdirectory, and files

I'm trying to make a script to list all directories, subdirectories, and files in a given directory. I tried this: import sys, os root = "/home/patate/directory/" path = os.path.join(root, "targetdirectory") for r, d, f in os.walk(path): for file in f: print(os.path.join(root, file)) Unfortunately, it doesn't work properly. I get all the files, but not their complete paths. For example, if the directory struct would be: /home/patate/directory/targetdirectory/123/456/789/file.txt It would print: /home/patate/directory/targetdirectory/file.txt I need the first result.

Output: Use os.path.join to concatenate the directory and file name: import os for path, subdirs, files in os.walk(root): for name in files: print(os.path.join(path, name)) Note the usage of path and not root in the concatenation, since using root would be incorrect. In Python 3.4, the pathlib module was added for easier path manipulations. So the equivalent to os.path.join would be: pathlib.PurePath(path, name) The advantage of pathlib is that you can use a variety of useful methods on paths. If you use the concrete Path variant you can also do actual OS calls through them, like changing into a directory, deleting the path, opening the file it points to and much more.


Instruction:
Get list of values for list of keys

Is there a built-in/quick way to use a list of keys to a dictionary to get a list of corresponding items? For instance I have: >>> mydict = {'one': 1, 'two': 2, 'three': 3} >>> mykeys = ['three', 'one'] How can I use mykeys to get the corresponding values in the dictionary as a list? >>> mydict.WHAT_GOES_HERE(mykeys) [3, 1]

Output: A list comprehension seems to be a good way to do this: >>> [mydict[x] for x in mykeys] [3, 1]


Instruction:
Accessing class variables from a list comprehension in the class definition

How do you access other class variables from a list comprehension within the class definition? The following works in Python 2 but fails in Python 3: class Foo: x = 5 y = [x for i in range(1)] Python 3.11 gives the error: NameError: name 'x' is not defined Trying Foo.x doesn't work either. Any ideas on how to do this in Python 3? A slightly more complicated motivating example: from collections import namedtuple class StateDatabase: State = namedtuple('State', ['name', 'capital']) db = [State(*args) for args in [ ['Alabama', 'Montgomery'], ['Alaska', 'Juneau'], # ... ]] In this example, apply() would have been a decent workaround, but it is sadly removed from Python 3.

Output: Class scope and list, set or dictionary comprehensions, as well as generator expressions, do not mix. The TL;DR You cannot access the class scope from functions, list comprehensions or generator expressions enclosed in that scope; they act as if that scope does not exist. In Python 2, list comprehensions were implemented using a shortcut so actually could access the class scope, but in Python 3 they got their own scope (as they should have had all along) and thus your example breaks. Other comprehension types have their own scope regardless of Python version, so a similar example with a set or dict comprehension would break in Python 2. # Same error, in Python 2 or 3 y = {x: x for i in range(1)} The why; or, the official word on this In Python 3, list comprehensions were given a proper scope (local namespace) of their own, to prevent their local variables bleeding over into the surrounding scope (see List comprehension rebinds names even after scope of comprehension. Is this right?). That's great when using such a list comprehension in a module or in a function, but in classes, scoping is a little, uhm, strange. This is documented in pep 227: Names in class scope are not accessible. Names are resolved in the innermost enclosing function scope. If a class definition occurs in a chain of nested scopes, the resolution process skips class definitions. and in the class compound statement documentation: The class’s suite is then executed in a new execution frame (see section Naming and binding), using a newly created local namespace and the original global namespace. (Usually, the suite contains only function definitions.) When the class’s suite finishes execution, its execution frame is discarded but its local namespace is saved. [4] A class object is then created using the inheritance list for the base classes and the saved local namespace for the attribute dictionary. Emphasis mine; the execution frame is the temporary scope. Because the scope is repurposed as the attributes on a class object, allowing it to be used as a nonlocal scope as well leads to undefined behaviour; what would happen if a class method referred to x as a nested scope variable, then manipulates Foo.x as well, for example? More importantly, what would that mean for subclasses of Foo? Python has to treat a class scope differently as it is very different from a function scope. Last, but definitely not least, the linked Naming and binding section in the Execution model documentation mentions class scopes explicitly: The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods – this includes comprehensions and generator expressions since they are implemented using a function scope. This means that the following will fail: class A: a = 42 b = list(a + i for i in range(10)) The (small) exception; or, why one part may still work There's one part of a comprehension or generator expression that executes in the surrounding scope, regardless of Python version. That would be the expression for the outermost iterable. In your example, it's the range(1): y = [x for i in range(1)] # ^^^^^^^^ Thus, using x in that expression would not throw an error: # Runs fine y = [i for i in range(x)] This only applies to the outermost iterable; if a comprehension has multiple for clauses, the iterables for inner for clauses are evaluated in the comprehension's scope: # NameError y = [i for i in range(1) for j in range(x)] # ^^^^^^^^^^^^^^^^^ ----------------- # outer loop inner, nested loop This design decision was made in order to throw an error at genexp creation time instead of iteration time when creating the outermost iterable of a generator expression throws an error, or when the outermost iterable turns out not to be iterable. Comprehensions share this behavior for consistency. Looking under the hood; or, way more detail than you ever wanted You can see this all in action using the dis module. I'm using Python 3.3 in the following examples, because it adds qualified names that neatly identify the code objects we want to inspect. The bytecode produced is otherwise functionally identical to Python 3.2. To create a class, Python essentially takes the whole suite that makes up the class body (so everything indented one level deeper than the class <name>: line), and executes that as if it were a function: >>> import dis >>> def foo(): ... class Foo: ... x = 5 ... y = [x for i in range(1)] ... return Foo ... >>> dis.dis(foo) 2 0 LOAD_BUILD_CLASS 1 LOAD_CONST 1 (<code object Foo at 0x10a436030, file "<stdin>", line 2>) 4 LOAD_CONST 2 ('Foo') 7 MAKE_FUNCTION 0 10 LOAD_CONST 2 ('Foo') 13 CALL_FUNCTION 2 (2 positional, 0 keyword pair) 16 STORE_FAST 0 (Foo) 5 19 LOAD_FAST 0 (Foo) 22 RETURN_VALUE The first LOAD_CONST there loads a code object for the Foo class body, then makes that into a function, and calls it. The result of that call is then used to create the namespace of the class, its __dict__. So far so good. The thing to note here is that the bytecode contains a nested code object; in Python, class definitions, functions, comprehensions and generators all are represented as code objects that contain not only bytecode, but also structures that represent local variables, constants, variables taken from globals, and variables taken from the nested scope. The compiled bytecode refers to those structures and the python interpreter knows how to access those given the bytecodes presented. The important thing to remember here is that Python creates these structures at compile time; the class suite is a code object (<code object Foo at 0x10a436030, file "<stdin>", line 2>) that is already compiled. Let's inspect that code object that creates the class body itself; code objects have a co_consts structure: >>> foo.__code__.co_consts (None, <code object Foo at 0x10a436030, file "<stdin>", line 2>, 'Foo') >>> dis.dis(foo.__code__.co_consts[1]) 2 0 LOAD_FAST 0 (__locals__) 3 STORE_LOCALS 4 LOAD_NAME 0 (__name__) 7 STORE_NAME 1 (__module__) 10 LOAD_CONST 0 ('foo.<locals>.Foo') 13 STORE_NAME 2 (__qualname__) 3 16 LOAD_CONST 1 (5) 19 STORE_NAME 3 (x) 4 22 LOAD_CONST 2 (<code object <listcomp> at 0x10a385420, file "<stdin>", line 4>) 25 LOAD_CONST 3 ('foo.<locals>.Foo.<listcomp>') 28 MAKE_FUNCTION 0 31 LOAD_NAME 4 (range) 34 LOAD_CONST 4 (1) 37 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 40 GET_ITER 41 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 44 STORE_NAME 5 (y) 47 LOAD_CONST 5 (None) 50 RETURN_VALUE The above bytecode creates the class body. The function is executed and the resulting locals() namespace, containing x and y is used to create the class (except that it doesn't work because x isn't defined as a global). Note that after storing 5 in x, it loads another code object; that's the list comprehension; it is wrapped in a function object just like the class body was; the created function takes a positional argument, the range(1) iterable to use for its looping code, cast to an iterator. As shown in the bytecode, range(1) is evaluated in the class scope. From this you can see that the only difference between a code object for a function or a generator, and a code object for a comprehension is that the latter is executed immediately when the parent code object is executed; the bytecode simply creates a function on the fly and executes it in a few small steps. Python 2.x uses inline bytecode there instead, here is output from Python 2.7: 2 0 LOAD_NAME 0 (__name__) 3 STORE_NAME 1 (__module__) 3 6 LOAD_CONST 0 (5) 9 STORE_NAME 2 (x) 4 12 BUILD_LIST 0 15 LOAD_NAME 3 (range) 18 LOAD_CONST 1 (1) 21 CALL_FUNCTION 1 24 GET_ITER >> 25 FOR_ITER 12 (to 40) 28 STORE_NAME 4 (i) 31 LOAD_NAME 2 (x) 34 LIST_APPEND 2 37 JUMP_ABSOLUTE 25 >> 40 STORE_NAME 5 (y) 43 LOAD_LOCALS 44 RETURN_VALUE No code object is loaded, instead a FOR_ITER loop is run inline. So in Python 3.x, the list generator was given a proper code object of its own, which means it has its own scope. However, the comprehension was compiled together with the rest of the python source code when the module or script was first loaded by the interpreter, and the compiler does not consider a class suite a valid scope. Any referenced variables in a list comprehension must look in the scope surrounding the class definition, recursively. If the variable wasn't found by the compiler, it marks it as a global. Disassembly of the list comprehension code object shows that x is indeed loaded as a global: >>> foo.__code__.co_consts[1].co_consts ('foo.<locals>.Foo', 5, <code object <listcomp> at 0x10a385420, file "<stdin>", line 4>, 'foo.<locals>.Foo.<listcomp>', 1, None) >>> dis.dis(foo.__code__.co_consts[1].co_consts[2]) 4 0 BUILD_LIST 0 3 LOAD_FAST 0 (.0) >> 6 FOR_ITER 12 (to 21) 9 STORE_FAST 1 (i) 12 LOAD_GLOBAL 0 (x) 15 LIST_APPEND 2 18 JUMP_ABSOLUTE 6 >> 21 RETURN_VALUE This chunk of bytecode loads the first argument passed in (the range(1) iterator), and just like the Python 2.x version uses FOR_ITER to loop over it and create its output. Had we defined x in the foo function instead, x would be a cell variable (cells refer to nested scopes): >>> def foo(): ... x = 2 ... class Foo: ... x = 5 ... y = [x for i in range(1)] ... return Foo ... >>> dis.dis(foo.__code__.co_consts[2].co_consts[2]) 5 0 BUILD_LIST 0 3 LOAD_FAST 0 (.0) >> 6 FOR_ITER 12 (to 21) 9 STORE_FAST 1 (i) 12 LOAD_DEREF 0 (x) 15 LIST_APPEND 2 18 JUMP_ABSOLUTE 6 >> 21 RETURN_VALUE The LOAD_DEREF will indirectly load x from the code object cell objects: >>> foo.__code__.co_cellvars # foo function `x` ('x',) >>> foo.__code__.co_consts[2].co_cellvars # Foo class, no cell variables () >>> foo.__code__.co_consts[2].co_consts[2].co_freevars # Refers to `x` in foo ('x',) >>> foo().y [2] The actual referencing looks the value up from the current frame data structures, which were initialized from a function object's .__closure__ attribute. Since the function created for the comprehension code object is discarded again, we do not get to inspect that function's closure. To see a closure in action, we'd have to inspect a nested function instead: >>> def spam(x): ... def eggs(): ... return x ... return eggs ... >>> spam(1).__code__.co_freevars ('x',) >>> spam(1)() 1 >>> spam(1).__closure__ >>> spam(1).__closure__[0].cell_contents 1 >>> spam(5).__closure__[0].cell_contents 5 So, to summarize: List comprehensions get their own code objects in Python 3 (up to Python 3.11), and there is no difference between code objects for functions, generators or comprehensions; comprehension code objects are wrapped in a temporary function object and called immediately. Code objects are created at compile time, and any non-local variables are marked as either global or as free variables, based on the nested scopes of the code. The class body is not considered a scope for looking up those variables. When executing the code, Python has only to look into the globals, or the closure of the currently executing object. Since the compiler didn't include the class body as a scope, the temporary function namespace is not considered. A workaround; or, what to do about it If you were to create an explicit scope for the x variable, like in a function, you can use class-scope variables for a list comprehension: >>> class Foo: ... x = 5 ... def y(x): ... return [x for i in range(1)] ... y = y(x) ... >>> Foo.y [5] The 'temporary' y function can be called directly; we replace it when we do with its return value. Its scope is considered when resolving x: >>> foo.__code__.co_consts[1].co_consts[2] <code object y at 0x10a5df5d0, file "<stdin>", line 4> >>> foo.__code__.co_consts[1].co_consts[2].co_cellvars ('x',) Of course, people reading your code will scratch their heads over this a little; you may want to put a big fat comment in there explaining why you are doing this. The best work-around is to just use __init__ to create an instance variable instead: def __init__(self): self.y = [self.x for i in range(1)] and avoid all the head-scratching, and questions to explain yourself. For your own concrete example, I would not even store the namedtuple on the class; either use the output directly (don't store the generated class at all), or use a global: from collections import namedtuple State = namedtuple('State', ['name', 'capital']) class StateDatabase: db = [State(*args) for args in [ ('Alabama', 'Montgomery'), ('Alaska', 'Juneau'), # ... ]] PEP 709; part of Python 3.12, changes some of this all again In Python 3.12, comprehensions have been made a lot more efficient by removing the nested function and inlining the loop, while still maintaining a separate scope. The details of how this was done are outlined in PEP 709 - Inlined comprehensions, but the long and short of it is that instead of creating a new function object and then calling it, with LOAD_CONST, MAKE_FUNCTION and CALL bytecodes, any clashing names used in the loop are first moved to the stack before executing the comprehension bytecode inline. It is important to note that this change only affects performance and interaction with the class scope has not changed. You still can't access names created in a class scope, for the reasons outlined above. Using Python 3.12.0b4 the bytecode for the Foo class now looks like this: # creating `def foo()` and its bytecode elided Disassembly of <code object Foo at 0x104e97000, file "<stdin>", line 2>: 2 0 RESUME 0 2 LOAD_NAME 0 (__name__) 4 STORE_NAME 1 (__module__) 6 LOAD_CONST 0 ('foo.<locals>.Foo') 8 STORE_NAME 2 (__qualname__) 3 10 LOAD_CONST 1 (5) 12 STORE_NAME 3 (x) 4 14 PUSH_NULL 16 LOAD_NAME 4 (range) 18 LOAD_CONST 2 (1) 20 CALL 1 28 GET_ITER 30 LOAD_FAST_AND_CLEAR 0 (.0) 32 LOAD_FAST_AND_CLEAR 1 (i) 34 LOAD_FAST_AND_CLEAR 2 (x) 36 SWAP 4 38 BUILD_LIST 0 40 SWAP 2 >> 42 FOR_ITER 8 (to 62) 46 STORE_FAST 1 (i) 48 LOAD_GLOBAL 6 (x) 58 LIST_APPEND 2 60 JUMP_BACKWARD 10 (to 42) >> 62 END_FOR 64 SWAP 4 66 STORE_FAST 2 (x) 68 STORE_FAST 1 (i) 70 STORE_FAST 0 (.0) 72 STORE_NAME 5 (y) 74 RETURN_CONST 3 (None) Here, the most important bytecode is the one at offset 34: 34 LOAD_FAST_AND_CLEAR 2 (x) This takes the value for the variable x in the local scope and pushes it on the stack, and then clears the name. If there is no variable x in the current scope, this stores a C NULL value on the stack. The name is now gone from the local scope now until the bytecode at offset 66 is reached: 66 STORE_FAST 2 (x) This restores x to what it was before the list comprehension; if a NULL was stored on the stack to indicate that there was no variable named x, then there still won't be a variable x after this bytecode has been executed. The rest of the bytecode between the LOAD_FAST_AND_CLEAR and STORE_FAST calls is more or less the same it was before, with SWAP bytecodes used to access the iterator for the range(1) object instead of LOAD_FAST (.0) in the function bytecode in earlier Python 3.x versions.


Instruction:
Pylint "unresolved import" error in Visual Studio Code

I am using the following setup macOS v10.14 (Mojave) Python 3.7.1 Visual Studio Code 1.30 Pylint 2.2.2 Django 2.1.4 I want to use linting to make my life a bit easier in Visual Studio Code. However, for every import I have states "unresolved import". Even on default Django imports (i.e. from django.db import models). I presume it is because it is not seeing the virtual environment Python files. Everything works just fine, but it's starting to get annoying. The interpreter choices I have are all system versions of Python. It does not seem to see my virtual environment Python at all (it is not in the same directory as my workspace, so that part makes sense). If I set up the python.PythonPath in the settings.json file, it just ignores it and does not list my virtual environment path as an option. I also tried setting it up in my global Python settings, but it also does not show up. Is there a quick fix to get it working?

Output: In your workspace settings, you can set your Python path like this: { "python.defaultInterpreterPath": "/path/to/your/venv/bin/python", }


Instruction:
How to convert hexadecimal string to bytes in Python?

I have a long Hex string that represents a series of values of different types. I need to convert this Hex String into bytes or bytearray so that I can extract each value from the raw data. How can I do this? For example, the string "ab" should convert to the bytes b"\xab" or equivalent byte array. Longer example: >>> # what to use in place of `convert` here? >>> convert("8e71c61de6a2321336184f813379ec6bf4a3fb79e63cd12b") b'\x8eq\xc6\x1d\xe6\xa22\x136\x18O\x813y\xeck\xf4\xa3\xfby\xe6<\xd1+'

Output: Suppose your hex string is something like >>> hex_string = "deadbeef" Convert it to a bytearray (Python 3 and 2.7): >>> bytearray.fromhex(hex_string) bytearray(b'\xde\xad\xbe\xef') Convert it to a bytes object (Python 3): >>> bytes.fromhex(hex_string) b'\xde\xad\xbe\xef' Note that bytes is an immutable version of bytearray. Convert it to a string (Python ≤ 2.7): >>> hex_data = hex_string.decode("hex") >>> hex_data "\xde\xad\xbe\xef"


Instruction:
Combining conda environment.yml with pip requirements.txt

I work with conda environments and need some pip packages as well, e.g. pre-compiled wheels from ~gohlke. At the moment I have two files: environment.yml for conda with: # run: conda env create --file environment.yml name: test-env dependencies: - python>=3.5 - anaconda and requirements.txt for pip which can be used after activating above conda environment: # run: pip install -i requirements.txt docx gooey http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl Is there a possibility to combine them in one file (for conda)?

Output: Pip dependencies can be included in the environment.yml file like this (docs): # run: conda env create --file environment.yml name: test-env dependencies: - python>=3.5 - anaconda - pip - numpy=1.13.3 # pin version for conda - pip: # works for regular pip packages - docx - gooey - matplotlib==2.0.0 # pin version for pip # and for wheels - http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl It also works for .whl files in the same directory (see Dengar's answer) as well as with common pip packages.


Instruction:
How does Django's nested Meta class work?

I am using Django which allows people to add extra parameters to a class by using class Meta. class FooModel(models.Model): ... class Meta: ... The only thing I found in Python's documentation was: class FooMetaClass(type): ... class FooClass: __metaclass__ = FooMetaClass However, I don't think this is the same thing.

Output: You are asking a question about two different things: Meta inner class in Django models: This is just a class container with some options (metadata) attached to the model. It defines such things as available permissions, associated database table name, whether the model is abstract or not, singular and plural versions of the name etc. Short explanation is here: Django docs: Models: Meta options List of available meta options is here: Django docs: Model Meta options For latest version of Django: Django docs: Model Meta options Metaclass in Python: The best description is here: What are metaclasses in Python?


Instruction:
Specifying and saving a figure with exact size in pixels

Say I have an image of size 3841 x 7195 pixels. I would like to save the contents of the figure to disk, resulting in an image of the exact size I specify in pixels. No axis, no titles. Just the image. I don't personally care about DPIs, as I only want to specify the size the image takes in the screen in disk in pixels. I have read other threads, and they all seem to do conversions to inches and then specify the dimensions of the figure in inches and adjust dpi's in some way. I would like to avoid dealing with the potential loss of accuracy that could result from pixel-to-inches conversions. I have tried with: w = 7195 h = 3841 fig = plt.figure(frameon=False) fig.set_size_inches(w,h) ax = plt.Axes(fig, [0., 0., 1., 1.]) ax.set_axis_off() fig.add_axes(ax) ax.imshow(im_np, aspect='normal') fig.savefig(some_path, dpi=1) with no luck (Python complains that width and height must each be below 32768 (?)) From everything I have seen, matplotlib requires the figure size to be specified in inches and dpi, but I am only interested in the pixels the figure takes in disk. How can I do this? To clarify: I am looking for a way to do this with matplotlib, and not with other image-saving libraries.

Output: Matplotlib doesn't work with pixels directly, but rather physical sizes and DPI. If you want to display a figure with a certain pixel size, you need to know the DPI of your monitor. For example this link will detect that for you. If you have an image of 3841x7195 pixels it is unlikely that you monitor will be that large, so you won't be able to show a figure of that size (matplotlib requires the figure to fit in the screen, if you ask for a size too large it will shrink to the screen size). Let's imagine you want an 800x800 pixel image just for an example. Here's how to show an 800x800 pixel image in my monitor (my_dpi=96): plt.figure(figsize=(800/my_dpi, 800/my_dpi), dpi=my_dpi) So you basically just divide the dimensions in pixels by your DPI. If you want to save a figure of a specific size, then it is a different matter. Screen DPIs are not so important anymore (unless you ask for a figure that won't fit in the screen). Using the same example of the 800x800 pixel figure, we can save it in different resolutions using the dpi keyword of savefig. To save it in the same resolution as the screen just use the same dpi: plt.savefig('my_fig.png', dpi=my_dpi) To save it as an 8000x8000 pixel image, use a dpi 10 times larger: plt.savefig('my_fig.png', dpi=my_dpi * 10) Note that the setting of the DPI is not supported by all backends. Here, the PNG backend is used, but the pdf and ps backends will implement the size differently. Also, changing the DPI and sizes will also affect things like fontsize. A larger DPI will keep the same relative sizes of fonts and elements, but if you want smaller fonts for a larger figure you need to increase the physical size instead of the DPI. Getting back to your example, if you want to save a image with 3841 x 7195 pixels, you could do the following: plt.figure(figsize=(3.841, 7.195), dpi=100) ( your code ...) plt.savefig('myfig.png', dpi=1000) Note that I used the figure dpi of 100 to fit in most screens, but saved with dpi=1000 to achieve the required resolution. In my system this produces a png with 3840x7190 pixels -- it seems that the DPI saved is always 0.02 pixels/inch smaller than the selected value, which will have a (small) effect on large image sizes. Some more discussion of this here.


Instruction:
Set up Python simpleHTTPserver on Windows

I want to set up Python SimpleHTTPServer on Windows XP. I have Python installed on my computer. I am executing the following command: python -m SimpleHTTPServer 8888 But I am getting the error: C:\Python33\python.exe: No module named SimpleHTTPServer Is SimpleHTTPServer for Python available on Windows? If yes, what do I do to set up the server?

Output: From Stack Overflow question What is the Python 3 equivalent of "python -m SimpleHTTPServer": SimpleHTTPServer is for python2, so you're getting the error. In python3, The following works: python -m http.server [<portNo>] Because using Python 3, the module SimpleHTTPServer has been replaced by http.server, at least in Windows.


Instruction:
Find unique rows in numpy.array

I need to find unique rows in a numpy.array. For example: >>> a # I have array([[1, 1, 1, 0, 0, 0], [0, 1, 1, 1, 0, 0], [0, 1, 1, 1, 0, 0], [1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 0]]) >>> new_a # I want to get to array([[1, 1, 1, 0, 0, 0], [0, 1, 1, 1, 0, 0], [1, 1, 1, 1, 1, 0]]) I know that i can create a set and loop over the array, but I am looking for an efficient pure numpy solution. I believe that there is a way to set data type to void and then I could just use numpy.unique, but I couldn't figure out how to make it work.

Output: As of NumPy 1.13, one can simply choose the axis for selection of unique values in any N-dim array. To get unique rows, use np.unique as follows: unique_rows = np.unique(original_array, axis=0)


Instruction:
Removing all non-numeric characters from string in Python

How do we remove all non-numeric characters from a string in Python?

Output: >>> import re >>> re.sub("[^0-9]", "", "sdkjh987978asd098as0980a98sd") '987978098098098' >>> # or >>> re.sub(r"\D", "", "sdkjh987978asd098as0980a98sd") '987978098098098'


Instruction:
Having options in argparse with a dash

I want to have some options in argparse module such as --pm-export however when I try to use it like args.pm-export I get the error that there is not attribute pm. How can I get around this issue? Is it possible to have - in command line options?

Output: From the argparse docs: For optional argument actions, the value of dest is normally inferred from the option strings. ArgumentParser generates the value of dest by taking the first long option string and stripping away the initial -- string. Any internal - characters will be converted to _ characters to make sure the string is a valid attribute name. So you should be using args.pm_export.


Instruction:
How can I time a code segment for testing performance with Pythons timeit?

I've a python script which works just as it should, but I need to write the execution time. I've googled that I should use timeit but I can't seem to get it to work. My Python script looks like this: import sys import getopt import timeit import random import os import re import ibm_db import time from string import maketrans myfile = open("results_update.txt", "a") for r in range(100): rannumber = random.randint(0, 100) update = "update TABLE set val = %i where MyCount >= '2010' and MyCount < '2012' and number = '250'" % rannumber #print rannumber conn = ibm_db.pconnect("dsn=myDB","usrname","secretPWD") for r in range(5): print "Run %s\n" % r ibm_db.execute(query_stmt) query_stmt = ibm_db.prepare(conn, update) myfile.close() ibm_db.close(conn) What I need is the time it takes to execute the query and write it to the file results_update.txt. The purpose is to test an update statement for my database with different indexes and tuning mechanisms.

Output: You can use time.time() or time.clock() before and after the block you want to time. import time t0 = time.time() code_block t1 = time.time() total = t1-t0 This method is not as exact as timeit (it does not average several runs) but it is straightforward. time.time() (in Windows and Linux) and time.clock() (in Linux) are not precise enough for fast functions because they return the time in an integer number of seconds (you'll get total = 0 for short durations). In this case or if you want to average the time elapsed by several runs, you have to manually call the function multiple times (As I think you already do in you example code and timeit does this automatically when you set its number argument) import time def myfast(): code n = 10000 t0 = time.time() for i in range(n): myfast() t1 = time.time() total_n = t1-t0 In Windows, time.clock() has higher precision because it returns the time in microseconds whereas time.time() returns the time in seconds. So choose whichever makes sense for your situation.


Instruction:
How to add hovering annotations to a plot

I am using matplotlib to make scatter plots. Each point on the scatter plot is associated with a named object. I would like to be able to see the name of an object when I hover my cursor over the point on the scatter plot associated with that object. In particular, it would be nice to be able to quickly see the names of the points that are outliers. The closest thing I have been able to find while searching here is the annotate command, but that appears to create a fixed label on the plot. Unfortunately, with the number of points that I have, the scatter plot would be unreadable if I labeled each point. Does anyone know of a way to create labels that only appear when the cursor hovers in the vicinity of that point?

Output: Here is a code that uses a scatter and shows an annotation upon hovering over the scatter points. import matplotlib.pyplot as plt import numpy as np; np.random.seed(1) x = np.random.rand(15) y = np.random.rand(15) names = np.array(list("ABCDEFGHIJKLMNO")) c = np.random.randint(1,5,size=15) norm = plt.Normalize(1,4) cmap = plt.cm.RdYlGn fig,ax = plt.subplots() sc = plt.scatter(x,y,c=c, s=100, cmap=cmap, norm=norm) annot = ax.annotate("", xy=(0,0), xytext=(20,20),textcoords="offset points", bbox=dict(boxstyle="round", fc="w"), arrowprops=dict(arrowstyle="->")) annot.set_visible(False) def update_annot(ind): pos = sc.get_offsets()[ind["ind"][0]] annot.xy = pos text = "{}, {}".format(" ".join(list(map(str,ind["ind"]))), " ".join([names[n] for n in ind["ind"]])) annot.set_text(text) annot.get_bbox_patch().set_facecolor(cmap(norm(c[ind["ind"][0]]))) annot.get_bbox_patch().set_alpha(0.4) def hover(event): vis = annot.get_visible() if event.inaxes == ax: cont, ind = sc.contains(event) if cont: update_annot(ind) annot.set_visible(True) fig.canvas.draw_idle() else: if vis: annot.set_visible(False) fig.canvas.draw_idle() fig.canvas.mpl_connect("motion_notify_event", hover) plt.show() Because people also want to use this solution for a line plot instead of a scatter, the following would be the same solution for plot (which works slightly differently). import matplotlib.pyplot as plt import numpy as np; np.random.seed(1) x = np.sort(np.random.rand(15)) y = np.sort(np.random.rand(15)) names = np.array(list("ABCDEFGHIJKLMNO")) norm = plt.Normalize(1,4) cmap = plt.cm.RdYlGn fig,ax = plt.subplots() line, = plt.plot(x,y, marker="o") annot = ax.annotate("", xy=(0,0), xytext=(-20,20),textcoords="offset points", bbox=dict(boxstyle="round", fc="w"), arrowprops=dict(arrowstyle="->")) annot.set_visible(False) def update_annot(ind): x,y = line.get_data() annot.xy = (x[ind["ind"][0]], y[ind["ind"][0]]) text = "{}, {}".format(" ".join(list(map(str,ind["ind"]))), " ".join([names[n] for n in ind["ind"]])) annot.set_text(text) annot.get_bbox_patch().set_alpha(0.4) def hover(event): vis = annot.get_visible() if event.inaxes == ax: cont, ind = line.contains(event) if cont: update_annot(ind) annot.set_visible(True) fig.canvas.draw_idle() else: if vis: annot.set_visible(False) fig.canvas.draw_idle() fig.canvas.mpl_connect("motion_notify_event", hover) plt.show() In case someone is looking for a solution for lines in twin axes, refer to How to make labels appear when hovering over a point in multiple axis? In case someone is looking for a solution for bar plots, please refer to e.g. this answer.


Instruction:
Pandas read_csv from url

I'm trying to read a csv-file from given URL, using Python 3.x: import pandas as pd import requests url = "https://github.com/cs109/2014_data/blob/master/countries.csv" s = requests.get(url).content c = pd.read_csv(s) I have the following error "Expected file path name or file-like object, got <class 'bytes'> type" How can I fix this? I'm using Python 3.4

Output: Update: From pandas 0.19.2 you can now just pass read_csv() the url directly, although that will fail if it requires authentication. For older pandas versions, or if you need authentication, or for any other HTTP-fault-tolerant reason: Use pandas.read_csv with a file-like object as the first argument. If you want to read the csv from a string, you can use io.StringIO. For the URL https://github.com/cs109/2014_data/blob/master/countries.csv, you get html response, not raw csv; you should use the url given by the Raw link in the github page for getting raw csv response , which is https://raw.githubusercontent.com/cs109/2014_data/master/countries.csv Example: import pandas as pd import io import requests url = "https://raw.githubusercontent.com/cs109/2014_data/master/countries.csv" s = requests.get(url).content c = pd.read_csv(io.StringIO(s.decode('utf-8'))) Note: in Python 2.x, the string-buffer object was StringIO.StringIO


Instruction:
How can I set up a virtual environment for Python in Visual Studio Code?

In my project folder I created a venv folder: python -m venv venv When I run command select python interpreter in Visual Studio Code, my venv folder is not shown. I went one level up like suggested here, but Visual Studio Code doesn't see my virtual interpreter. What did I miss?

Output: I have been using Visual Studio Code for a while now and found an another way to show virtual environments in Visual Studio Code. Go to the parent folder in which venv is there through a command prompt. Type code . and Enter. [It is working on both Windows and Linux for me.] That should also show the virtual environments present in that folder. Original Answer I almost run into same problem every time I am working on Visual Studio Code using venv. I follow the below steps: Go to menu File → Preferences → Settings. Click on Workspace settings. Under Files:Association, in the JSON: Schemas section, you will find Edit in settings.json. Click on that. Update "python.defaultInterpreterPath": "Your_venv_path/bin/python" under workspace settings. (For Windows): Update "python.defaultInterpreterPath": "Your_venv_path\Scripts\python.exe" under workspace settings. Restart Visual Studio Code in case if it still doesn't show your venv. Note: Use python.pythonPath instead of python.defaultInterpreterPath for older versions.


Instruction:
How can I read inputs as numbers?

Why are x and y strings instead of ints in the below code? (Note: in Python 2.x use raw_input(). In Python 3.x use input(). raw_input() was renamed to input() in Python 3.x) play = True while play: x = input("Enter a number: ") y = input("Enter a number: ") print(x + y) print(x - y) print(x * y) print(x / y) print(x % y) if input("Play again? ") == "no": play = False

Output: Since Python 3, input returns a string which you have to explicitly convert to int, like this x = int(input("Enter a number: ")) y = int(input("Enter a number: ")) You can accept numbers of any base like this >>> data = int(input("Enter a number: "), 8) Enter a number: 777 >>> data 511 >>> data = int(input("Enter a number: "), 16) Enter a number: FFFF >>> data 65535 >>> data = int(input("Enter a number: "), 2) Enter a number: 10101010101 >>> data 1365 The second parameter tells it the base of the number and then internally it understands and converts it. If the entered data is wrong it will throw a ValueError. >>> data = int(input("Enter a number: "), 2) Enter a number: 1234 Traceback (most recent call last): File "<input>", line 1, in <module> ValueError: invalid literal for int() with base 2: '1234' For values that can have a fractional component, the type would be float rather than int: x = float(input("Enter a number:")) Differences between Python 2 and 3 Summary Python 2's input function evaluated the received data, converting it to an integer implicitly (read the next section to understand the implication), but Python 3's input function does not do that anymore. Python 2's equivalent of Python 3's input is the raw_input function. Python 2.x There were two functions to get user input, called input and raw_input. The difference between them is, raw_input doesn't evaluate the data and returns as it is, in string form. But, input will evaluate whatever you entered and the result of evaluation will be returned. For example, >>> import sys >>> sys.version '2.7.6 (default, Mar 22 2014, 22:59:56) \n[GCC 4.8.2]' >>> data = input("Enter a number: ") Enter a number: 5 + 17 >>> data, type(data) (22, <type 'int'>) The data 5 + 17 is evaluated and the result is 22. When it evaluates the expression 5 + 17, it detects that you are adding two numbers and so the result will also be of the same int type. So, the type conversion is done for free, and 22 is returned as the result of the input and stored in the data variable. You can think of input as the raw_input composed with an eval call. >>> data = eval(raw_input("Enter a number: ")) Enter a number: 5 + 17 >>> data, type(data) (22, <type 'int'>) Note: You should be careful when you are using input in Python 2.x. I explained why one should be careful when using it, in this answer. But, raw_input doesn't evaluate the input and returns as it is, as a string. >>> import sys >>> sys.version '2.7.6 (default, Mar 22 2014, 22:59:56) \n[GCC 4.8.2]' >>> data = raw_input("Enter a number: ") Enter a number: 5 + 17 >>> data, type(data) ('5 + 17', <type 'str'>) Python 3.x Python 3.x's input and Python 2.x's raw_input are similar and raw_input is not available in Python 3.x. >>> import sys >>> sys.version '3.4.0 (default, Apr 11 2014, 13:05:11) \n[GCC 4.8.2]' >>> data = input("Enter a number: ") Enter a number: 5 + 17 >>> data, type(data) ('5 + 17', <class 'str'>)


Instruction:
How to suppress or capture the output of subprocess.run()?

From the examples in docs on subprocess.run() it seems like there shouldn't be any output from subprocess.run(["ls", "-l"]) # doesn't capture output However, when I try it in a python shell the listing gets printed. I wonder if this is the default behaviour and how to suppress the output of run().

Output: Suppressing Here is how to suppress output, in order of decreasing levels of cleanliness. They assume you are on Python 3. You can redirect to the special subprocess.DEVNULL target. import subprocess # To redirect stdout (only): subprocess.run( ['ls', '-l'], stdout = subprocess.DEVNULL ) # to redirect stderr to /dev/null as well: subprocess.run( ['ls', '-l'], stdout = subprocess.DEVNULL, stderr = subprocess.DEVNULL ) # Alternatively, you can merge stderr and stdout streams and redirect # the one stream to /dev/null subprocess.run( ['ls', '-l'], stdout = subprocess.DEVNULL, stderr = subprocess.STDOUT ) If you want a fully manual method, can redirect to /dev/null by opening the file handle yourself. Everything else would be identical to method #1. import os import subprocess with open(os.devnull, 'w') as devnull: subprocess.run( ['ls', '-l'], stdout = devnull ) Capturing Here is how to capture output (to use later or parse), in order of decreasing levels of cleanliness. They assume you are on Python 3. NOTE: The below examples use universal_newlines=True (Python <= 3.6). This causes the STDOUT and STDERR to be captured as str instead of bytes. Omit universal_newlines=True to get bytes data Python >= 3.7 accepts text=True as a short form for universal_newlines=True If you simply want to capture both STDOUT and STDERR independently, AND you are on Python >= 3.7, use capture_output=True. import subprocess result = subprocess.run( ['ls', '-l'], capture_output = True, # Python >= 3.7 only text = True # Python >= 3.7 only ) print(result.stdout) print(result.stderr) You can use subprocess.PIPE to capture STDOUT and STDERR independently. This works on any version of Python that supports subprocess.run. import subprocess result = subprocess.run( ['ls', '-l'], stdout = subprocess.PIPE, universal_newlines = True # Python >= 3.7 also accepts "text=True" ) print(result.stdout) # To also capture stderr... result = subprocess.run( ['ls', '-l'], stdout = subprocess.PIPE, stderr = subprocess.PIPE, universal_newlines = True # Python >= 3.7 also accepts "text=True" ) print(result.stdout) print(result.stderr) # To mix stdout and stderr into a single string result = subprocess.run( ['ls', '-l'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT, universal_newlines = True # Python >= 3.7 also accepts "text=True" ) print(result.stdout)


Instruction:
Check if a word is in a string in Python

I'm working with Python, and I'm trying to find out if you can tell if a word is in a string. I have found some information about identifying if the word is in the string - using .find, but is there a way to do an if statement. I would like to have something like the following: if string.find(word): print("success")

Output: What is wrong with: if word in mystring: print('success')


Instruction:
Whether to use apply vs transform on a group object, to subtract two columns and get mean

Consider the following dataframe: columns = ['A', 'B', 'C', 'D'] records = [ ['foo', 'one', 0.162003, 0.087469], ['bar', 'one', -1.156319, -1.5262719999999999], ['foo', 'two', 0.833892, -1.666304], ['bar', 'three', -2.026673, -0.32205700000000004], ['foo', 'two', 0.41145200000000004, -0.9543709999999999], ['bar', 'two', 0.765878, -0.095968], ['foo', 'one', -0.65489, 0.678091], ['foo', 'three', -1.789842, -1.130922] ] df = pd.DataFrame.from_records(records, columns=columns) """ A B C D 0 foo one 0.162003 0.087469 1 bar one -1.156319 -1.526272 2 foo two 0.833892 -1.666304 3 bar three -2.026673 -0.322057 4 foo two 0.411452 -0.954371 5 bar two 0.765878 -0.095968 6 foo one -0.654890 0.678091 7 foo three -1.789842 -1.130922 """ The following commands work: df.groupby('A').apply(lambda x: (x['C'] - x['D'])) df.groupby('A').apply(lambda x: (x['C'] - x['D']).mean()) but none of the following work: df.groupby('A').transform(lambda x: (x['C'] - x['D'])) # KeyError or ValueError: could not broadcast input array from shape (5) into shape (5,3) df.groupby('A').transform(lambda x: (x['C'] - x['D']).mean()) # KeyError or TypeError: cannot concatenate a non-NDFrame object Why? The example on the documentation seems to suggest that calling transform on a group allows one to do row-wise operation processing: # Note that the following suggests row-wise operation (x.mean is the column mean) zscore = lambda x: (x - x.mean()) / x.std() transformed = ts.groupby(key).transform(zscore) In other words, I thought that transform is essentially a specific type of apply (the one that does not aggregate). Where am I wrong? For reference, below is the construction of the original dataframe above: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'foo'], 'B' : ['one', 'one', 'two', 'three', 'two', 'two', 'one', 'three'], 'C' : randn(8), 'D' : randn(8)})

Output: Two major differences between apply and transform There are two major differences between the transform and apply groupby methods. Input: apply implicitly passes all the columns for each group as a DataFrame to the custom function. while transform passes each column for each group individually as a Series to the custom function. Output: The custom function passed to apply can return a scalar, or a Series or DataFrame (or numpy array or even list). The custom function passed to transform must return a sequence (a one dimensional Series, array or list) the same length as the group. So, transform works on just one Series at a time and apply works on the entire DataFrame at once. Inspecting the custom function It can help quite a bit to inspect the input to your custom function passed to apply or transform. Examples Let's create some sample data and inspect the groups so that you can see what I am talking about: import pandas as pd import numpy as np df = pd.DataFrame({'State':['Texas', 'Texas', 'Florida', 'Florida'], 'a':[4,5,1,3], 'b':[6,10,3,11]}) State a b 0 Texas 4 6 1 Texas 5 10 2 Florida 1 3 3 Florida 3 11 Let's create a simple custom function that prints out the type of the implicitly passed object and then raises an exception so that execution can be stopped. def inspect(x): print(type(x)) raise Now let's pass this function to both the groupby apply and transform methods to see what object is passed to it: df.groupby('State').apply(inspect) <class 'pandas.core.frame.DataFrame'> <class 'pandas.core.frame.DataFrame'> RuntimeError As you can see, a DataFrame is passed into the inspect function. You might be wondering why the type, DataFrame, got printed out twice. Pandas runs the first group twice. It does this to determine if there is a fast way to complete the computation or not. This is a minor detail that you shouldn't worry about. Now, let's do the same thing with transform df.groupby('State').transform(inspect) <class 'pandas.core.series.Series'> <class 'pandas.core.series.Series'> RuntimeError It is passed a Series - a totally different Pandas object. So, transform is only allowed to work with a single Series at a time. It is impossible for it to act on two columns at the same time. So, if we try and subtract column a from b inside of our custom function we would get an error with transform. See below: def subtract_two(x): return x['a'] - x['b'] df.groupby('State').transform(subtract_two) KeyError: ('a', 'occurred at index a') We get a KeyError as pandas is attempting to find the Series index a which does not exist. You can complete this operation with apply as it has the entire DataFrame: df.groupby('State').apply(subtract_two) State Florida 2 -2 3 -8 Texas 0 -2 1 -5 dtype: int64 The output is a Series and a little confusing as the original index is kept, but we have access to all columns. Displaying the passed pandas object It can help even more to display the entire pandas object within the custom function, so you can see exactly what you are operating with. You can use print statements by I like to use the display function from the IPython.display module so that the DataFrames get nicely outputted in HTML in a jupyter notebook: from IPython.display import display def subtract_two(x): display(x) return x['a'] - x['b'] Screenshot: Transform must return a single dimensional sequence the same size as the group The other difference is that transform must return a single dimensional sequence the same size as the group. In this particular instance, each group has two rows, so transform must return a sequence of two rows. If it does not then an error is raised: def return_three(x): return np.array([1, 2, 3]) df.groupby('State').transform(return_three) ValueError: transform must return a scalar value for each group The error message is not really descriptive of the problem. You must return a sequence the same length as the group. So, a function like this would work: def rand_group_len(x): return np.random.rand(len(x)) df.groupby('State').transform(rand_group_len) a b 0 0.962070 0.151440 1 0.440956 0.782176 2 0.642218 0.483257 3 0.056047 0.238208 Returning a single scalar object also works for transform If you return just a single scalar from your custom function, then transform will use it for each of the rows in the group: def group_sum(x): return x.sum() df.groupby('State').transform(group_sum) a b 0 9 16 1 9 16 2 4 14 3 4 14


Instruction:
Given parallel lists, how can I sort one while permuting (rearranging) the other in the same way?

Suppose I have: list1 = [3, 2, 4, 1, 1] list2 = ['three', 'two', 'four', 'one', 'one2'] Calling list1.sort() will sort it, resulting in [1, 1, 2, 3, 4]. However, can I get list2 to be rearranged in sync with that, to get a result like this? list1 = [1, 1, 2, 3, 4] list2 = ['one', 'one2', 'two', 'three', 'four'] Sometimes, people phrase the problem differently: given two lists, they would like to use one to determine the sort order for the other - i.e., sort list2 in the order described by the corresponding values in list1. The trick is that this is equivalent to sorting the "key" values (list1), and then rearranging list2 in the same way. In other words, exactly what is described here. Some answers for the other question, though, discard the "sorted keys" afterwards. See also: How can I sort a list, according to where its elements appear in another list? - this is another common way that people want to sort one list "based on" another. Before attempting to close duplicate questions, take special care to check exactly what the OP wants. A key clue: do the lists need to be the same length?

Output: One classic approach to this problem is to use the "decorate, sort, undecorate" idiom, which is especially simple using python's built-in zip function: >>> list1 = [3,2,4,1, 1] >>> list2 = ['three', 'two', 'four', 'one', 'one2'] >>> list1, list2 = zip(*sorted(zip(list1, list2))) >>> list1 (1, 1, 2, 3, 4) >>> list2 ('one', 'one2', 'two', 'three', 'four') These of course are no longer lists, but that's easily remedied, if it matters: >>> list1, list2 = (list(t) for t in zip(*sorted(zip(list1, list2)))) >>> list1 [1, 1, 2, 3, 4] >>> list2 ['one', 'one2', 'two', 'three', 'four'] It's worth noting that the above may sacrifice speed for terseness; the in-place version, which takes up 3 lines, is a tad faster on my machine for small lists: >>> %timeit zip(*sorted(zip(list1, list2))) 100000 loops, best of 3: 3.3 us per loop >>> %timeit tups = zip(list1, list2); tups.sort(); zip(*tups) 100000 loops, best of 3: 2.84 us per loop On the other hand, for larger lists, the one-line version could be faster: >>> %timeit zip(*sorted(zip(list1, list2))) 100 loops, best of 3: 8.09 ms per loop >>> %timeit tups = zip(list1, list2); tups.sort(); zip(*tups) 100 loops, best of 3: 8.51 ms per loop As Quantum7 points out, JSF's suggestion is a bit faster still, but it will probably only ever be a little bit faster, because Python uses the very same DSU idiom internally for all key-based sorts. It's just happening a little closer to the bare metal. (This shows just how well optimized the zip routines are!) I think the zip-based approach is more flexible and is a little more readable, so I prefer it. Note that when elements of list1 are equal, this approach will end up comparing elements of list2. If elements of list2 don't support comparison, or don't produce a boolean when compared (for example, if list2 is a list of NumPy arrays), this will fail, and if elements of list2 are very expensive to compare, it might be better to avoid comparison anyway. In that case, you can sort indices as suggested in jfs's answer, or you can give the sort a key function that avoids comparing elements of list2: result1, result2 = zip(*sorted(zip(list1, list2), key=lambda x: x[0])) Also, the use of zip(*...) as a transpose fails when the input is empty. If your inputs might be empty, you will have to handle that case separately.


Instruction:
Subclass in type hinting

I want to allow type hinting using Python 3 to accept sub classes of a certain class. E.g.: class A: pass class B(A): pass class C(A): pass def process_any_subclass_type_of_A(cls: A): if cls == B: # do something elif cls == C: # do something else Now when typing the following code: process_any_subclass_type_of_A(B) I get an PyCharm IDE hint Expected type A, got Type[B] instead. How can I change type hinting here to accept any subtypes of A? According to PEP 484 ("Expressions whose type is a subtype of a specific argument type are also accepted for that argument."), I understand that my solution (cls: A) should work?

Output: When you specify cls: A, you're saying that cls expects an instance of type A. For python 3.5.2 through 3.8, the type hint to specify cls as a class object for the type A (or its subtypes) uses typing.Type. from typing import Type def process_any_subclass_type_of_A(cls: Type[A]): pass From The type of class objects: Sometimes you want to talk about class objects that inherit from a given class. This can be spelled as Type[C] where C is a class. In other words, when C is the name of a class, using C to annotate an argument declares that the argument is an instance of C (or of a subclass of C), but using Type[C] as an argument annotation declares that the argument is a class object deriving from C (or C itself). From python 3.9 onwards, it is recommended to use the builtin type instead. def process_any_subclass_type_of_A(cls: type[A]): pass


Instruction:
How to check a string for specific characters?

How can I check if a string has several specific characters in it using Python 2? For example, given the following string: The criminals stole $1,000,000 in jewels. How do I detect if it has dollar signs ($), commas (,), and numbers?

Output: Assuming your string is s: '$' in s # found '$' not in s # not found # original answer given, but less Pythonic than the above... s.find('$')==-1 # not found s.find('$')!=-1 # found And so on for other characters. ... or pattern = re.compile(r'[\d\$,]') if pattern.findall(s): print('Found') else: print('Not found') ... or chars = set('0123456789$,') if any((c in chars) for c in s): print('Found') else: print('Not Found')


Instruction:
Why does Python 3 round half to even?

I was just re-reading What’s New In Python 3.0 and it states: The round() function rounding strategy and return type have changed. Exact halfway cases are now rounded to the nearest even result instead of away from zero. (For example, round(2.5) now returns 2 rather than 3.) and the documentation for round(): For the built-in types supporting round(), values are rounded to the closest multiple of 10 to the power minus n; if two multiples are equally close, rounding is done toward the even choice So, in Python 2 (for example, v2.7.3) I get the expected: round(2.5) 3.0 round(3.5) 4.0 However, now under Python 3 (for example v3.2.3): round(2.5) 2 round(3.5) 4 This seems counter-intuitive and contrary to what I understand about rounding (and bound to trip up people). English isn't my native language but until I read this I thought I knew what rounding meant :-/ I am sure at the time Python 3 was introduced there must have been some discussion of this, but I was unable to find a good reason in my search. Does anyone have insight into why this was changed to this? Are there any other mainstream programming languages (e.g., C, C++, Java, Perl, ..) that do this sort of (to me inconsistent) rounding? What am I missing here? UPDATE: @Li-aungYip's comment re "Banker's rounding" gave me the right search term/keywords to search for and I found this SO question: Why does .NET use banker's rounding as default?, so I will be reading that carefully.

Output: Python 3's way (called "round half to even" or "banker's rounding") is considered the standard rounding method these days, though some language implementations aren't on the bus yet. The simple "always round 0.5 up" technique results in a slight bias toward the higher number. With large numbers of calculations, this can be significant. The Python 3 approach eliminates this issue. There is more than one method of rounding in common use. IEEE 754, the international standard for floating-point math, defines five different rounding methods (the one used by Python 3 is the default). And there are others. This behavior is not as widely known as it ought to be. AppleScript was, if I remember correctly, an early adopter of this rounding method. The round command in AppleScript offers several options, but round-toward-even is the default as it is in IEEE 754. Apparently the engineer who implemented the round command got so fed up with all the requests to "make it work like I learned in school" that he implemented just that: round 2.5 rounding as taught in school is a valid AppleScript command. :-)


Instruction:
Multiprocessing : use tqdm to display a progress bar

To make my code more "pythonic" and faster, I use multiprocessing and a map function to send it a) the function and b) the range of iterations. The implanted solution (i.e., calling tqdm directly on the range tqdm.tqdm(range(0, 30))) does not work with multiprocessing (as formulated in the code below). The progress bar is displayed from 0 to 100% (when python reads the code?) but it does not indicate the actual progress of the map function. How can one display a progress bar that indicates at which step the 'map' function is ? from multiprocessing import Pool import tqdm import time def _foo(my_number): square = my_number * my_number time.sleep(1) return square if __name__ == '__main__': p = Pool(2) r = p.map(_foo, tqdm.tqdm(range(0, 30))) p.close() p.join() Any help or suggestions are welcome...

Output: Solution found. Be careful! Due to multiprocessing, the estimation time (iteration per loop, total time, etc.) could be unstable, but the progress bar works perfectly. Note: Context manager for Pool is only available in Python 3.3+. import time from multiprocessing import Pool from random import randint from tqdm import tqdm def _foo(my_number): square = my_number * my_number time.sleep(randint(1, 2) / 2) return square if __name__ == "__main__": max_ = 30 with Pool(processes=2) as p, tqdm(total=max_) as pbar: for result in p.imap(_foo, range(0, max_)): pbar.update() pbar.refresh() # do something with `result`


Instruction:
How to set class attribute with await in __init__

How can I define a class with await in the constructor or class body? For example what I want: import asyncio # some code class Foo(object): async def __init__(self, settings): self.settings = settings self.pool = await create_pool(dsn) foo = Foo(settings) # it raises: # TypeError: __init__() should return None, not 'coroutine' or example with class body attribute: class Foo(object): self.pool = await create_pool(dsn) # Sure it raises syntax Error def __init__(self, settings): self.settings = settings foo = Foo(settings) My solution (But I would like to see a more elegant way) class Foo(object): def __init__(self, settings): self.settings = settings async def init(self): self.pool = await create_pool(dsn) foo = Foo(settings) await foo.init()

Output: Most magic methods aren't designed to work with async def/await - in general, you should only be using await inside the dedicated asynchronous magic methods - __aiter__, __anext__, __aenter__, and __aexit__. Using it inside other magic methods either won't work at all, as is the case with __init__ (unless you use some tricks described in other answers here), or will force you to always use whatever triggers the magic method call in an asynchronous context. Existing asyncio libraries tend to deal with this in one of two ways: First, I've seen the factory pattern used (asyncio-redis, for example): import asyncio dsn = "..." class Foo(object): @classmethod async def create(cls, settings): self = cls() self.settings = settings self.pool = await create_pool(dsn) return self async def main(settings): settings = "..." foo = await Foo.create(settings) Other libraries use a top-level coroutine function that creates the object, rather than a factory method: import asyncio dsn = "..." async def create_foo(settings): foo = Foo(settings) await foo._init() return foo class Foo(object): def __init__(self, settings): self.settings = settings async def _init(self): self.pool = await create_pool(dsn) async def main(): settings = "..." foo = await create_foo(settings) The create_pool function from aiopg that you want to call in __init__ is actually using this exact pattern. This at least addresses the __init__ issue. I haven't seen class variables that make asynchronous calls in the wild that I can recall, so I don't know that any well-established patterns have emerged.


Instruction:
if A vs if A is not None:

Can I use: if A: instead of if A is not None: The latter seems so verbose. Is there a difference?

Output: The statement if A: will call A.__bool__() (see Special method names documentation), which was called __nonzero__ in Python 2, and use the return value of that function. Here's the summary: object.__bool__(self) Called to implement truth value testing and the built-in operation bool(); should return False or True. When this method is not defined, __len__() is called, if it is defined, and the object is considered true if its result is nonzero. If a class defines neither __len__() nor __bool__(), all its instances are considered true. On the other hand, if A is not None: compares only the reference A with None to see whether it is the same or not.


Instruction:
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings

I was trying to download a GUI, but the terminal kept giving me this error: Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases. I'm trying to install it using this command: python -m pip install --upgrade pip setuptools virtualenv

Output: You need to download Python from https://python.org. When in the installation, be sure to check the option that adds Python to PATH.