Instruction:
Why is my poetry virtualenv using the system python instead of the pyenv python that I set?

I've recently installed both Pyenv and Poetry and want to create a new Python 3.8 project. I've set both the global and local versions of python to 3.8.1 using the appropriate Pyenv commands (pyenv global 3.8.1 for example). When I run pyenv version in my terminal the output is 3.8.1. as expected. Now, the problem is that when I create a new python project with Poetry (poetry new my-project), the generated pyproject.toml file creates a project with python 2.7: [tool.poetry] name = "my-project" version = "0.1.0" description = "" authors = ["user <user@email.com>"] [tool.poetry.dependencies] python = "^2.7" [tool.poetry.dev-dependencies] pytest = "^4.6" [build-system] requires = ["poetry>=0.12"] build-backend = "poetry.masonry.api" It seems that Poetry defaults back to the system version of Python. How do I change this so that it uses the version installed with Pyenv? Edit I'm using MacOS, which comes bundled with Python 2.7. I think that might be causing some of the issues here. I've reinstalled Python 3.8 again with Pyenv, but when I hit Poetry install I get the following error: The currently activated Python version 2.7.16 is not supported by the project (^3.8). Trying to find and use a compatible version. [NoCompatiblePythonVersionFound] Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command. Should I create an environment explicitly for the project using Pyenv or should the project be able to access the correct Python version after running pyenv local 3.8.1.? When I do the latter, nothing changes and I still get the same errors.

Output: Alright, I figured the problem. A little embarrassingly, I had not run pyenv shell 3.8.1 before running any of the other commands. Everything works now. Thank you all for your efforts.


Instruction:
Prepend element to numpy array

I have the following numpy array import numpy as np X = np.array([[5.], [4.], [3.], [2.], [1.]]) I want to insert [6.] at the beginning. I've tried: X = X.insert(X, 0) how do I insert into X?

Output: numpy has an insert function that's accesible via np.insert with documentation. You'll want to use it in this case like so: X = np.insert(X, 0, 6., axis=0) the first argument X specifies the object to be inserted into. The second argument 0 specifies where. The third argument 6. specifies what is to be inserted. The fourth argument axis=0 specifies that the insertion should happen at position 0 for every column. We could've chosen rows but your X is a columns vector, so I figured we'd stay consistent.


Instruction:
Django SUM Query?

I have a query akin to the following: SELECT SUM(name) FROM table WHERE name IS NULL How does that SUM translate into a QuerySet in Django? i.e. What operation xyz does it translate to, in something like MyModel.objects.xyz()?

Output: Update: The following incorporates the ISNULL aspect of the original query: from django.db.models import Sum ModelName.objects.filter(field_name__isnull=True).aggregate(Sum('field_name')) # returns {'field_name__sum': 1000} for example You're looking for the Sum aggregation function, which works as follows: ModelName.objects.aggregate(Sum('field_name')) See: https://docs.djangoproject.com/en/dev/ref/models/querysets/#sum


Instruction:
Creating lowpass filter in SciPy - understanding methods and units

I am trying to filter a noisy heart rate signal with python. Because heart rates should never be above about 220 beats per minute, I want to filter out all noise above 220 bpm. I converted 220/minute into 3.66666666 Hertz and then converted that Hertz to rad/s to get 23.0383461 rad/sec. The sampling frequency of the chip that takes data is 30Hz so I converted that to rad/s to get 188.495559 rad/s. After looking up some stuff online I found some functions for a bandpass filter that I wanted to make into a lowpass. Here is the link the bandpass code, so I converted it to be this: from scipy.signal import butter, lfilter from scipy.signal import freqs def butter_lowpass(cutOff, fs, order=5): nyq = 0.5 * fs normalCutoff = cutOff / nyq b, a = butter(order, normalCutoff, btype='low', analog = True) return b, a def butter_lowpass_filter(data, cutOff, fs, order=4): b, a = butter_lowpass(cutOff, fs, order=order) y = lfilter(b, a, data) return y cutOff = 23.1 #cutoff frequency in rad/s fs = 188.495559 #sampling frequency in rad/s order = 20 #order of filter #print sticker_data.ps1_dxdt2 y = butter_lowpass_filter(data, cutOff, fs, order) plt.plot(y) I am very confused by this though because I am pretty sure the butter function takes in the cutoff and sampling frequency in rad/s but I seem to be getting a weird output. Is it actually in Hz? Secondly, what is the purpose of these two lines: nyq = 0.5 * fs normalCutoff = cutOff / nyq I know it's something about normalization but I thought the nyquist was 2 times the sampling requency, not one half. And why are you using the nyquist as a normalizer? Can someone explain more about how to create filters with these functions? I plotted the filter using: w, h = signal.freqs(b, a) plt.plot(w, 20 * np.log10(abs(h))) plt.xscale('log') plt.title('Butterworth filter frequency response') plt.xlabel('Frequency [radians / second]') plt.ylabel('Amplitude [dB]') plt.margins(0, 0.1) plt.grid(which='both', axis='both') plt.axvline(100, color='green') # cutoff frequency plt.show() and got this which clearly does not cut-off at 23 rad/s:

Output: A few comments: The Nyquist frequency is half the sampling rate. You are working with regularly sampled data, so you want a digital filter, not an analog filter. This means you should not use analog=True in the call to butter, and you should use scipy.signal.freqz (not freqs) to generate the frequency response. One goal of those short utility functions is to allow you to leave all your frequencies expressed in Hz. You shouldn't have to convert to rad/sec. As long as you express your frequencies with consistent units, the fs parameter of the SciPy functions will take care of the scaling for you. Here's my modified version of your script, followed by the plot that it generates. import numpy as np from scipy.signal import butter, lfilter, freqz import matplotlib.pyplot as plt def butter_lowpass(cutoff, fs, order=5): return butter(order, cutoff, fs=fs, btype='low', analog=False) def butter_lowpass_filter(data, cutoff, fs, order=5): b, a = butter_lowpass(cutoff, fs, order=order) y = lfilter(b, a, data) return y # Filter requirements. order = 6 fs = 30.0 # sample rate, Hz cutoff = 3.667 # desired cutoff frequency of the filter, Hz # Get the filter coefficients so we can check its frequency response. b, a = butter_lowpass(cutoff, fs, order) # Plot the frequency response. w, h = freqz(b, a, fs=fs, worN=8000) plt.subplot(2, 1, 1) plt.plot(w, np.abs(h), 'b') plt.plot(cutoff, 0.5*np.sqrt(2), 'ko') plt.axvline(cutoff, color='k') plt.xlim(0, 0.5*fs) plt.title("Lowpass Filter Frequency Response") plt.xlabel('Frequency [Hz]') plt.grid() # Demonstrate the use of the filter. # First make some data to be filtered. T = 5.0 # seconds n = int(T * fs) # total number of samples t = np.linspace(0, T, n, endpoint=False) # "Noisy" data. We want to recover the 1.2 Hz signal from this. data = np.sin(1.2*2*np.pi*t) + 1.5*np.cos(9*2*np.pi*t) + 0.5*np.sin(12.0*2*np.pi*t) # Filter the data, and plot both the original and filtered signals. y = butter_lowpass_filter(data, cutoff, fs, order) plt.subplot(2, 1, 2) plt.plot(t, data, 'b-', label='data') plt.plot(t, y, 'g-', linewidth=2, label='filtered data') plt.xlabel('Time [sec]') plt.grid() plt.legend() plt.subplots_adjust(hspace=0.35) plt.show()


Instruction:
How do I tell a Python script to use a particular version

How do I, in the main.py module (presumably), tell Python which interpreter to use? What I mean is: if I want a particular script to use version 3 of Python to interpret the entire program, how do I do that? Bonus: How would this affect a virtualenv? Am I right in thinking that if I create a virtualenv for my program and then tell it to use a different version of Python, will I encounter some conflicts?

Output: You can add a shebang line the to the top of the script: #!/usr/bin/env python2.7 But that will only work when executing as ./my_program.py. If you execute as python my_program.py, then the whatever Python version that which python returns will be used. In re: to virtualenv use: virtualenv -p /usr/bin/python3.2 or whatever to set it up to use that Python executable.


Instruction:
How to use values stored in variables as case patterns?

I'm trying to understand the new structural pattern matching syntax in Python 3.10. I understand that it is possible to match on literal values like this: def handle(retcode): match retcode: case 200: print('success') case 404: print('not found') case _: print('unknown') handle(404) # not found However, if I refactor and move these values to module-level variables, it results in an error because the statements now represent structures or patterns rather than values: SUCCESS = 200 NOT_FOUND = 404 def handle(retcode): match retcode: case SUCCESS: print('success') case NOT_FOUND: print('not found') case _: print('unknown') handle(404) # File "<ipython-input-2-fa4ae710e263>", line 6 # case SUCCESS: # ^ # SyntaxError: name capture 'SUCCESS' makes remaining patterns unreachable Is there any way to use the match statement to match values that are stored within variables?

Output: If the constant you're testing against is a dotted name, then it should be treated as a constant instead of as the name of the variable to put the capture in (see PEP 636 # Matching against constants and enums): class Codes: SUCCESS = 200 NOT_FOUND = 404 def handle(retcode): match retcode: case Codes.SUCCESS: print('success') case Codes.NOT_FOUND: print('not found') case _: print('unknown') Although, given how python is trying to implement pattern-matching, I think that for situations like this it's probably safer and clearer code to just use an if/elif/else tower when checking against constant values.


Instruction:
How can I plot NaN values as a special color with imshow?

I am trying to use imshow in matplotlib to plot data as a heatmap, but some of the values are NaNs. I'd like the NaNs to be rendered as a special color not found in the colormap. example: import numpy as np import matplotlib.pyplot as plt f = plt.figure() ax = f.add_subplot(111) a = np.arange(25).reshape((5,5)).astype(float) a[3,:] = np.nan ax.imshow(a, interpolation='nearest') f.canvas.draw() The resultant image is unexpectedly all blue (the lowest color in the jet colormap). However, if I do the plotting like this: ax.imshow(a, interpolation='nearest', vmin=0, vmax=24) --then I get something better, but the NaN values are drawn the same color as vmin... Is there a graceful way that I can set NaNs to be drawn with a special color (eg: gray or transparent)?

Output: With newer versions of Matplotlib, it is not necessary to use a masked array anymore. For example, let’s generate an array where every 7th value is a NaN: arr = np.arange(100, dtype=float).reshape(10, 10) arr[~(arr % 7).astype(bool)] = np.nan .cm.get_cmap() is replaced by .colormaps.get_cmap('viridis') in matplotlib v3.7.0 Set the color with .set_bad. import matplotlib as mpl import matplotlib.pyplot as plt import numpy as np arr = np.arange(100, dtype=float).reshape(10, 10) arr[~(arr % 7).astype(bool)] = np.nan cmap = mpl.colormaps.get_cmap('viridis') # viridis is the default colormap for imshow cmap.set_bad(color='red') plt.imshow(arr, cmap=cmap) .cm.get_cmap() is deprecated We can modify the current colormap and plot the array with the following lines: current_cmap = mpl.cm.get_cmap() current_cmap.set_bad(color='red') plt.imshow(arr)


Instruction:
Is it possible to run function in a subprocess without threading or writing a separate file/script.

import subprocess def my_function(x): return x + 100 output = subprocess.Popen(my_function, 1) #I would like to pass the function object and its arguments print output #desired output: 101 I have only found documentation on opening subprocesses using separate scripts. Does anyone know how to pass function objects or even an easy way to pass function code?

Output: I think you're looking for something more like the multiprocessing module: http://docs.python.org/library/multiprocessing.html#the-process-class The subprocess module is for spawning processes and doing things with their input/output - not for running functions. Here is a multiprocessing version of your code: from multiprocessing import Process, Queue # must be a global function def my_function(q, x): q.put(x + 100) if __name__ == '__main__': queue = Queue() p = Process(target=my_function, args=(queue, 1)) p.start() p.join() # this blocks until the process terminates result = queue.get() print result


Instruction:
Is there go up line character? (Opposite of \n)

I would like to overwrite something on a line above in a serial console. Is there a character that allows me to move up?

Output: Most terminals understand ANSI escape codes. The relevant "Control Sequence Introducer" commands for this use case: "\033[F" – move cursor to the beginning of the previous line "\033[A" – move cursor up one line Example (Python): print("\033[FMy text overwriting the previous line.")


Instruction:
Dynamically create Literal alias from list of valid values

I have a function which validates its argument to accept only values from a given list of valid options. Typing-wise, I reflect this behavior using a Literal type alias, like so: from typing import Literal VALID_ARGUMENTS = ['foo', 'bar'] Argument = Literal['foo', 'bar'] def func(argument: 'Argument') -> None: if argument not in VALID_ARGUMENTS: raise ValueError( f'argument must be one of {VALID_ARGUMENTS}' ) # ... This is a violation of the DRY principle, because I have to rewrite the list of valid arguments in the definition of my Literal type, even if it is already stored in the variable VALID_ARGUMENTS. How can I create the Argument Literal type dynamically, given the VALID_ARGUMENTS variable? The following things do not work: from typing import Literal, Union, NewType Argument = Literal[*VALID_ARGUMENTS] # SyntaxError: invalid syntax Argument = Literal[VALID_ARGUMENTS] # Parameters to generic types must be types Argument = Literal[Union[VALID_ARGUMENTS]] # TypeError: Union[arg, ...]: each arg must be a type. Got ['foo', 'bar']. Argument = NewType( 'Argument', Union[ Literal[valid_argument] for valid_argument in VALID_ARGUMENTS ] ) # Expected type 'Type[_T]', got 'list' instead Can it be done at all?

Output: Go the other way around, and build VALID_ARGUMENTS from Argument: Argument = typing.Literal['foo', 'bar'] VALID_ARGUMENTS: typing.Tuple[Argument, ...] = typing.get_args(Argument) I've used a tuple for VALID_ARGUMENTS here, but if for some reason you really prefer a list, you can get one: VALID_ARGUMENTS: typing.List[Argument] = list(typing.get_args(Argument)) It's possible at runtime to build Argument from VALID_ARGUMENTS, but doing so is incompatible with static analysis, which is the primary use case of type annotations. Doing so is also considered semantically invalid - the spec forbids parameterizing Literal with dynamically computed parameters. The runtime implementation simply doesn't have the information it would need to validate this. Building VALID_ARGUMENTS from Argument is the way to go.


Instruction:
Elegant way to create empty pandas DataFrame with NaN of type float

I want to create a Pandas DataFrame filled with NaNs. During my research I found an answer: import pandas as pd df = pd.DataFrame(index=range(0,4),columns=['A']) This code results in a DataFrame filled with NaNs of type "object". So they cannot be used later on for example with the interpolate() method. Therefore, I created the DataFrame with this complicated code (inspired by this answer): import pandas as pd import numpy as np dummyarray = np.empty((4,1)) dummyarray[:] = np.nan df = pd.DataFrame(dummyarray) This results in a DataFrame filled with NaN of type "float", so it can be used later on with interpolate(). Is there a more elegant way to create the same result?

Output: Simply pass the desired value as first argument, like 0, math.inf or, here, np.nan. The constructor then initializes and fills the value array to the size specified by arguments index and columns: >>> import numpy as np >>> import pandas as pd >>> df = pd.DataFrame(np.nan, index=[0, 1, 2, 3], columns=['A', 'B']) >>> df A B 0 NaN NaN 1 NaN NaN 2 NaN NaN 3 NaN NaN >>> df.dtypes A float64 B float64 dtype: object


Instruction:
Is it safe to replace '==' with 'is' to compare Boolean-values

I did several Boolean Comparisons: >>> (True or False) is True True >>> (True or False) == True True It sounds like == and is are interchangeable for Boolean-values. Sometimes it's more clear to use is I want to know that: Are True and False pre-allocated in python? Is bool(var) always return the same True(or False) with the pre-allocated True(or False)? Is it safe to replace == with is to compare Boolean-values? It's not about Best-Practice. I just want to know the Truth.

Output: It's rare to need to directly compare booleans. If you are doing something like: if some_bool == True: ... ...just change it to: if some_bool: ... No is or == needed. As commenters have pointed out, there are valid reasons to compare booleans. If both booleans are unknown and you want to know if one is equal to the other, you should use == or != rather than is or is not (the reason is explained below). Note that this is logically equivalent to xnor and xor respectively, which don't exist as logical operators in Python. Internally, there should only ever be two boolean literal objects (see also the C API), and bool(x) is True should be True if bool(x) == True for any Python program. Two caveats: This does not mean that x is True if x == True, however (eg. x = 1). This is true for the usual implementation of Python (CPython) but might not be true in other implementations. Hence == is a more reliable comparison.


Instruction:
How to set the range of y-axis for a seaborn boxplot

From the official seaborn documentation, I learned that you can create a boxplot as below: import seaborn as sns sns.set_style("whitegrid") tips = sns.load_dataset("tips") ax = sns.boxplot(x="day", y="total_bill", data=tips) My question is: how do I limit the range of y-axis of this plot? For example, I want the y-axis to be within [10, 40]. Is there any easy way to do this?

Output: It is standard matplotlib.pyplot: import matplotlib.pyplot as plt plt.ylim(10, 40) Or simpler, as mwaskom comments below: ax.set(ylim=(10, 40))


Instruction:
Opposite of melt in python pandas

I cannot figure out how to do "reverse melt" using Pandas in python. This is my starting data label type value 0 x a 1 1 x b 2 2 x c 3 3 y a 4 4 y b 5 5 y c 6 6 z a 7 7 z b 8 8 z c 9 This is the output I would like to have: label a b c x 1 2 3 y 4 5 6 z 7 8 9 I'm sure there is an easy way to do this, but I don't know how.

Output: there are a few ways: using .pivot: >>> origin.pivot(index='label', columns='type')['value'] type a b c label x 1 2 3 y 4 5 6 z 7 8 9 [3 rows x 3 columns] using pivot_table: >>> origin.pivot_table(values='value', index='label', columns='type') value type a b c label x 1 2 3 y 4 5 6 z 7 8 9 [3 rows x 3 columns] or .groupby followed by .unstack: >>> origin.groupby(['label', 'type'])['value'].aggregate('mean').unstack() type a b c label x 1 2 3 y 4 5 6 z 7 8 9 [3 rows x 3 columns]


Instruction:
Python MySQL connector - unread result found when using fetchone

I am inserting JSON data into a MySQL database I am parsing the JSON and then inserting it into a MySQL db using the python connector Through trial, I can see the error is associated with this piece of code for steps in result['routes'][0]['legs'][0]['steps']: query = ('SELECT leg_no FROM leg_data WHERE travel_mode = %s AND Orig_lat = %s AND Orig_lng = %s AND Dest_lat = %s AND Dest_lng = %s AND time_stamp = %s') if steps['travel_mode'] == "pub_tran": travel_mode = steps['travel_mode'] Orig_lat = steps['var_1']['dep']['lat'] Orig_lng = steps['var_1']['dep']['lng'] Dest_lat = steps['var_1']['arr']['lat'] Dest_lng = steps['var_1']['arr']['lng'] time_stamp = leg['_sent_time_stamp'] if steps['travel_mode'] =="a_pied": query = ('SELECT leg_no FROM leg_data WHERE travel_mode = %s AND Orig_lat = %s AND Orig_lng = %s AND Dest_lat = %s AND Dest_lng = %s AND time_stamp = %s') travel_mode = steps['travel_mode'] Orig_lat = steps['var_2']['lat'] Orig_lng = steps['var_2']['lng'] Dest_lat = steps['var_2']['lat'] Dest_lng = steps['var_2']['lng'] time_stamp = leg['_sent_time_stamp'] cursor.execute(query,(travel_mode, Orig_lat, Orig_lng, Dest_lat, Dest_lng, time_stamp)) leg_no = cursor.fetchone()[0] print(leg_no) I have inserted higher level details and am now searching the database to associate this lower level information with its parent. The only way to find this unique value is to search via the origin and destination coordinates with the time_stamp. I believe the logic is sound and by printing the leg_no immediately after this section, I can see values which appear at first inspection to be correct However, when added to the rest of the code, it causes subsequent sections where more data is inserted using the cursor to fail with this error - raise errors.InternalError("Unread result found.") mysql.connector.errors.InternalError: Unread result found. The issue seems similar to MySQL Unread Result with Python Is the query too complex and needs splitting or is there another issue? If the query is indeed too complex, can anyone advise how best to split this? EDIT As per @Gord's help, Ive tried to dump any unread results cursor.execute(query,(leg_travel_mode, leg_Orig_lat, leg_Orig_lng, leg_Dest_lat, leg_Dest_lng)) leg_no = cursor.fetchone()[0] try: cursor.fetchall() except mysql.connector.errors.InterfaceError as ie: if ie.msg == 'No result set to fetch from.': pass else: raise cursor.execute(query,(leg_travel_mode, leg_Orig_lat, leg_Orig_lng, leg_Dest_lat, leg_Dest_lng, time_stamp)) But, I still get raise errors.InternalError("Unread result found.") mysql.connector.errors.InternalError: Unread result found. [Finished in 3.3s with exit code 1] scratches head EDIT 2 - when I print the ie.msg, I get - No result set to fetch from

Output: All that was required was for buffered to be set to true! cursor = cnx.cursor(buffered=True) The reason is that without a buffered cursor, the results are "lazily" loaded, meaning that "fetchone" actually only fetches one row from the full result set of the query. When you will use the same cursor again, it will complain that you still have n-1 results (where n is the result set amount) waiting to be fetched. However, when you use a buffered cursor the connector fetches ALL rows behind the scenes and you just take one from the connector so the mysql db won't complain.


Instruction:
Consistently create same random numpy array

I am waiting for another developer to finish a piece of code that will return an np array of shape (100,2000) with values of either -1,0, or 1. In the meantime, I want to randomly create an array of the same characteristics so I can get a head start on my development and testing. The thing is that I want this randomly created array to be the same each time, so that I'm not testing against an array that keeps changing its value each time I re-run my process. I can create my array like this, but is there a way to create it so that it's the same each time. I can pickle the object and unpickle it, but wondering if there's another way. r = np.random.randint(3, size=(100, 2000)) - 1

Output: Simply seed the random number generator with a fixed value, e.g. numpy.random.seed(42) This way, you'll always get the same random number sequence. This function will seed the global default random number generator, and any call to a function in numpy.random will use and alter its state. This is fine for many simple use cases, but it's a form of global state with all the problems global state brings. For a cleaner solution, see Robert Kern's answer below.


Instruction:
ModuleNotFoundError: No module named 'virtualenv.seed.embed.via_app_data' when I created new env by virtualenv

I was creating a new virtual environment on Ubuntu 20.04: $ virtualenv my_env But it gave an error: ModuleNotFoundError: No module named 'virtualenv.seed.embed.via_app_data' Other info: $ virtualenv --version virtualenv 20.0.17 from /usr/lib/python3/dist-packages/virtualenv/__init__.py

Output: Try to create the virtual environment using directly venv module python3 -m venv my_env


Instruction:
ModuleNotFoundError with pytest

I want my tests folder separate to my application code. My project structure is like so myproject/ myproject/ myproject.py moduleone.py tests/ myproject_test.py myproject.py from moduleone import ModuleOne class MyProject(object) .... myproject_test.py from myproject.myproject import MyProject import pytest ... I use myproject.myproject since I use the command python -m pytest from the project root directory ./myproject/ However, then the imports within those modules fail with E ModuleNotFoundError: No module named 'moduleone' I am running Python 3.7 and have read that since 3.3, empty __init__ files are no longer needed which means my project becomes an implicit namespace package However, I have tried adding an __init__.py file in myproject/myproject/ and also tried adding a conftest.py file in myproject/ but neither works I have read answers that say to mess with the paths and then upvoted comments in other questions saying not to. What is the correct way and what am I missing? EDIT; Possibly related, I used a requirements.txt to install pytest using pip. Could this be related? And if so, what is the correct way to install pytest in this case? EDIT 2: One of the paths in sys.path is /usr/src/app/ which is a docker volume lined to /my/local/path/myproject/. Should the volume be /my/local/path/myproject/myproject/ instead?

Output: In 2023.02, according to the document of pytest, you can simply add following config to your pyproject.toml to solve this problem [tool.pytest.ini_options] pythonpath = "src" addopts = [ "--import-mode=importlib", ]


Instruction:
How to update() a single model instance retrieved by get() on Django ORM?

I have a function which currently calls Models.object.get(), which returns either 0 or 1 model objects: if it returns 0, I create a new model instance in the except DoesNotExist clause of the function. Otherwise, I would like to update the fields in the pre-existing instance, without creating a new one. I was originally attempting to call .update() on the instance which was found, but .update() seems to be only callable on a QuerySets. How do I get around changing a dozen fields, without calling .filter() and comparing the lengths to know if I have to create or update a pre-existing instance?

Output: With the advent of Django 1.7, there is now a new update_or_create QuerySet method, which should do exactly what you want. Just be careful of potential race conditions if uniqueness is not enforced at the database level. Example from the documentation: obj, created = Person.objects.update_or_create( first_name='John', last_name='Lennon', defaults={'first_name': 'Bob'}, ) The update_or_create method tries to fetch an object from database based on the given kwargs. If a match is found, it updates the fields passed in the defaults dictionary. Pre-Django 1.7: Change the model field values as appropriate, then call .save() to persist the changes: try: obj = Model.objects.get(field=value) obj.field = new_value obj.save() except Model.DoesNotExist: obj = Model.objects.create(field=new_value) # do something else with obj if need be


Instruction:
in python how do I convert a single digit number into a double digits string?

So say I have a = 5 I want to print it as a string '05'

Output: In python 3.6, the fstring or "formatted string literal" mechanism was introduced. f"{a:02}" is the equivalent of the .format format below, but a little bit more terse. python 3 before 3.6 prefers a somewhat more verbose formatting system: "{0:0=2d}".format(a) You can take shortcuts here, the above is probably the most verbose variant. The full documentation is available here: http://docs.python.org/3/library/string.html#string-formatting print "%02d"%a is the python 2 variant The relevant doc link for python2 is: http://docs.python.org/2/library/string.html#format-specification-mini-language


Instruction:
How to do unit testing of functions writing files using Python's 'unittest'

I have a Python function that writes an output file to disk. I want to write a unit test for it using Python's unittest module. How should I assert equality of files? I would like to get an error if the file content differs from the expected one + list of differences. As in the output of the Unix diff command. Is there an official or recommended way of doing that?

Output: The simplest thing is to write the output file, then read its contents, read the contents of the gold (expected) file, and compare them with simple string equality. If they are the same, delete the output file. If they are different, raise an assertion. This way, when the tests are done, every failed test will be represented with an output file, and you can use a third-party tool to diff them against the gold files (Beyond Compare is wonderful for this). If you really want to provide your own diff output, remember that the Python stdlib has the difflib module. The new unittest support in Python 3.1 includes an assertMultiLineEqual method that uses it to show diffs, similar to this: def assertMultiLineEqual(self, first, second, msg=None): """Assert that two multi-line strings are equal. If they aren't, show a nice diff. """ self.assertTrue(isinstance(first, str), 'First argument is not a string') self.assertTrue(isinstance(second, str), 'Second argument is not a string') if first != second: message = ''.join(difflib.ndiff(first.splitlines(True), second.splitlines(True))) if msg: message += " : " + msg self.fail("Multi-line strings are unequal:\n" + message)


Instruction:
Get Traceback of warnings

In numpy we can do np.seterr(invalid='raise') to get a traceback for warnings raising an error instead (see this post). Is there a general way for tracing warnings? Can I make python to give a traceback, when a warning is raised?

Output: You can get what you want by assigning to warnings.showwarning. The warnings module documentation itself recommends that you do that, so it's not that you're being tempted by the dark side of the source. :) You may replace this function with an alternative implementation by assigning to warnings.showwarning. You can define a new function that does what warning.showwarning normaly does and additionally it prints the stack. Then you place it instead of the original: import traceback import warnings import sys def warn_with_traceback(message, category, filename, lineno, file=None, line=None): log = file if hasattr(file,'write') else sys.stderr traceback.print_stack(file=log) log.write(warnings.formatwarning(message, category, filename, lineno, line)) warnings.showwarning = warn_with_traceback After this, every warning will print the stack trace as well as the warning message. Take into account, however, that if the warning is ignored because it is not the first one, nothing will happen, so you still need to execute: warnings.simplefilter("always") You can get a similar control that the one numpy.seterr gives through the warning module's filters If what you want is python to report every a warning every time it is triggered and not only the first time, you can include something like: import warnings warnings.simplefilter("always") You can get other behaviours by passing different strings as arguments. Using the same function you can also specify different behaviours for warnings depending on the module that raised them, the message they provide, the warning class, the line of code that is causing it and so on... You can check the list in the module documentation As an example, you can set all the warnings to raise exceptions, except the DeprecationWarnings that should be ignored completely: import warnings warnings.simplefilter("error") warnings.simplefilter("ignore", DeprecationWarning) This way you get the full traceback for each warning raised as error (only the first one, since execution will stop... but you can address them one by one, and create a filter to ignore the ones you don't want to hear about again...


Instruction:
Check if string has date, any format

How do I check if a string can be parsed to a date? Jan 19, 1990 January 19, 1990 Jan 19,1990 01/19/1990 01/19/90 1990 Jan 1990 January1990 These are all valid dates. If there's any concern regarding the lack of space in between stuff in item #3 and the last item above, that can be easily remedied via automatically inserting a space in between letters/characters and numbers, if so needed. But first, the basics: I tried putting it in an if statement: if datetime.strptime(item, '%Y') or datetime.strptime(item, '%b %d %y') or datetime.strptime(item, '%b %d %Y') or datetime.strptime(item, '%B %d %y') or datetime.strptime(item, '%B %d %Y'): But that's in a try-except block, and keeps returning something like this: 16343 time data 'JUNE1890' does not match format '%Y' Unless, it met the first condition in the if statement. To clarify, I don't actually need the value of the date - I just want to know if it is. Ideally, it would've been something like this: if item is date: print date else: print "Not a date" Is there any way to do this?

Output: The parse function in dateutils.parser is capable of parsing many date string formats to a datetime object. pip install python-dateutil If you simply want to know whether a particular string could represent or contain a valid date, you could try the following simple function: from dateutil.parser import parse def is_date(string, fuzzy=False): """ Return whether the string can be interpreted as a date. :param string: str, string to check for date :param fuzzy: bool, ignore unknown tokens in string if True """ try: parse(string, fuzzy=fuzzy) return True except ValueError: return False Then you have: >>> is_date("1990-12-1") True >>> is_date("2005/3") True >>> is_date("Jan 19, 1990") True >>> is_date("today is 2019-03-27") False >>> is_date("today is 2019-03-27", fuzzy=True) True >>> is_date("Monday at 12:01am") True >>> is_date("xyz_not_a_date") False >>> is_date("yesterday") False Custom parsing parse might recognise some strings as dates which you don't want to treat as dates. For example: Parsing "12" and "1999" will return a datetime object representing the current date with the day and year substituted for the number in the string "23, 4" and "23 4" will be parsed as datetime.datetime(2023, 4, 16, 0, 0). "Friday" will return the date of the nearest Friday in the future. Similarly "August" corresponds to the current date with the month changed to August. Also parse is not locale aware, so does not recognise months or days of the week in languages other than English. Both of these issues can be addressed to some extent by using a custom parserinfo class, which defines how month and day names are recognised: from dateutil.parser import parserinfo class CustomParserInfo(parserinfo): # three months in Spanish for illustration MONTHS = [("Enero", "Enero"), ("Feb", "Febrero"), ("Marzo", "Marzo")] An instance of this class can then be used with parse: >>> parse("Enero 1990") # ValueError: Unknown string format >>> parse("Enero 1990", parserinfo=CustomParserInfo()) datetime.datetime(1990, 1, 27, 0, 0)


Instruction:
Opposite of set.intersection in python?

In Python you can use a.intersection(b) to find the items common to both sets. Is there a way to do the disjoint opposite version of this? Items that are not common to both a and b; the unique items in a unioned with the unique items in b?

Output: You are looking for the symmetric difference; all elements that appear only in set a or in set b, but not both: a.symmetric_difference(b) From the set.symmetric_difference() method documentation: Return a new set with elements in either the set or other but not both. You can use the ^ operator too, if both a and b are sets: a ^ b while set.symmetric_difference() takes any iterable for the other argument. The output is the equivalent of (a | b) - (a & b), the union of both sets minus the intersection of both sets. Producing the output takes O(M+N) time for sets of length M and N, respectively; M steps to copy set a then N steps to alter that set based on each value in b: def symmetric_difference(a, b): result = set(a) for elem in b: try: result.remove(elem) except KeyError: result.add(elem) return result There are in-place variants too, where set a is altered directly; use a.symmetric_difference_update(b) or a ^= b. The in-place variant takes O(N) time, so it depends on the size of set b only: def symmetric_difference_update(a, b): for elem in b: try: a.remove(elem) except KeyError: a.add(elem) # no return, a has been updated in-place


Instruction:
NLTK python error: "TypeError: 'dict_keys' object is not subscriptable"

I'm following instructions for a class homework assignment and I'm supposed to look up the top 200 most frequently used words in a text file. Here's the last part of the code: fdist1 = FreqDist(NSmyText) vocab=fdist1.keys() vocab[:200] But when I press enter after the vocab 200 line, it returns: Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'dict_keys' object is not subscriptable Any suggestions on how to fix this so it can correctly return an answer?

Output: Looks like you are using Python 3. In Python 3 dict.keys() returns an iterable but not indexable object. The most simple (but not so efficient) solution would be: vocab = list(fdist1.keys())[:200] In some situations it is desirable to continue working with an iterator object instead of a list. This can be done with itertools.islice(): import itertools vocab_iterator = itertools.islice(fdist1.keys(), 200)


Instruction:
Django reverse lookup of foreign keys

I have a venue, this venue has many events happening there. My models look like this: class Venue(models.Model): title = models.CharField(max_length=200) date_published = models.DateTimeField('published date',default=datetime.now, blank=True) venue_latitude = models.CharField(max_length=200) venue_longitude = models.CharField(max_length=200) venue_address = models.CharField(max_length=200) venue_city = models.CharField(max_length=200) venue_state = models.CharField(max_length=200) venue_country = models.CharField(max_length=200) description = models.TextField() def __unicode__(self): return u'%s' % (self.title) class Event(models.Model): title = models.CharField(max_length=200) date_published = models.DateTimeField('published date',default=datetime.now, blank=True) date_start = models.DateTimeField('start date') date_end = models.DateTimeField('end date') def __unicode__(self): return self.title description = models.TextField() price = models.IntegerField(null=True, blank=True) venue = models.ForeignKey(Venue) I'd like to display all the events that are happening at a certain venue. How can I do that? My current view looks like: def detail(request, venue_id): venue = get_object_or_404(Venue, pk=venue_id) return render(request, 'venue-detail.html', {'venue': venue})

Output: You can use events = venue.event_set to go the other way. Note that venue.event_set is a manager object, like Event.objects, so you can call .all, .filter, .exclude and similar on it to get a queryset. See the Django documentation


Instruction:
How to get color of most recent plotted line in Python's plt

I plot a line without specifying the color (think: plt.plot(x,y)). Say the color comes out blue. Question: How do I obtain this color from the plt object so that I can put it into a variable? Seems like this is close (and potentially the solution): p = plt.plot(x,y) color = p[0].get_color() Updated question: I am not sure I understand the "0" index: Does p[0] always access the most recent plotted line?

Output: In your example, p is a list of Line2D object. In that example you have only one line object, p[0]. The following is an example plotting three lines. As more line is added, it is appended to the p. So if you want the color of the last plot, it will be p[-1].get_color(). import numpy as np import matplotlib.pyplot as plt x = np.arange(10) y = np.arange(10) p = plt.plot(x,y, x,y*2, x,y*3) # make three line plots type(p) # list type(p[0]) # <class 'matplotlib.lines.Line2D'> p[0].get_color() # 'b' p[1].get_color() # 'g' p[2].get_color() # 'r'


Instruction:
Mocking Functions Using Python Mock

I am trying to Mock a function (that returns some external content) using the python mock module. I'm having some trouble mocking functions that are imported into a module. For example, in util.py I have def get_content(): return "stuff" I want to mock util.get_content so that it returns something else. I am trying this: util.get_content=Mock(return_value="mocked stuff") If get_content gets invoked inside another module, it never actually seems to return the mocked object. Am I missing something in terms of how to use Mock? Note that if I invoke the following, things work correctly: >>> util.get_content=Mock(return_value="mocked stuff") >>> util.get_content() "mocked stuff" However, if get_content is called from inside another module, it invokes the original function instead of the mocked version: >>> from mymodule import MyObj >>> util.get_content=Mock(return_value="mocked stuff") >>> m=MyObj() >>> m.func() "stuff" Contents of mymodule.py from util import get_content class MyObj: def func(): get_content() So I guess my question is - how do I get invoke the Mocked version of a function from inside a module that I call? It appears that the from module import function may be to blame here, in that it doesn't point to the Mocked function.

Output: I think I have a workaround, though it's still not quite clear on how to solve the general case In mymodule, if I replace from util import get_content class MyObj: def func(): get_content() with import util class MyObj: def func(): util.get_content() The Mock seems to get invoked. It looks like the namespaces need to match (which makes sense). However, the weird thing is that I would expect import mymodule mymodule.get_content = mock.Mock(return_value="mocked stuff") to do the trick in the original case where I am using the from/import syntax (which now pulls in get_content into mymodule). But this still refers to the unmocked get_content. Turns out the namespace matters - just need to keep that in mind when writing your code.


Instruction:
A logarithmic colorbar in matplotlib scatter plot

I would like to make the colors of the points on the scatter plot correspond to the value of the void fraction, but on a logarithmic scale to amplify differences. I did this, but now when I do plt.colorbar(), it displays the log of the void fraction, when I really want the actual void fraction. How can I make a log scale on the colorbar with the appropriate labels of the void fraction, which belongs to [0.00001,1]? Here is an image of the plot I have now, but the void fraction colorbar is not appropriately labeled to correspond to the true void fraction, instead of the log of it. fig = plt.figure() plt.scatter(x,y,edgecolors='none',s=marker_size,c=np.log(void_fraction)) plt.colorbar() plt.title('Colorbar: void fraction') Thanks for your help.

Output: There is now a section of the documentation describing how color mapping and normalization works The way that matplotlib does color mapping is in two steps, first a Normalize function (wrapped up by the sub-classes of matplotlib.colors.Normalize) which maps the data you hand in to [0, 1]. The second step maps values in [0,1] -> RGBA space. You just need to use the LogNorm normalization class, passed in with the norm kwarg. plt.scatter(x,y,edgecolors='none',s=marker_size,c=void_fraction, norm=matplotlib.colors.LogNorm()) When you want to scale/tweak data for plotting, it is better to let matplotlib do the transformations than to do it your self. Normalize doc LogNorm doc matplotlib.color doc


Instruction:
Python functions call by reference

In some languages you can pass a parameter by reference or value by using a special reserved word like ref or val. When you pass a parameter to a Python function it never alters the value of the parameter on leaving the function.The only way to do this is by using the global reserved word (or as i understand it currently). Example 1: k = 2 def foo (n): n = n * n #clarity regarding comment below square = n return square j = foo(k) print j print k would show >>4 >>2 showing k to be unchanged. In this example the variable n is never changed Example 2: n = 0 def foo(): global n n = n * n return n In this example the variable n is changed. Is there any way in Python to call a function and tell Python that the parameter is either a value or reference parameter instead of using global?

Output: You can not change an immutable object, like str or tuple, inside a function in Python, but you can do things like: def foo(y): y[0] = y[0]**2 x = [5] foo(x) print x[0] # prints 25 That is a weird way to go about it, however, unless you need to always square certain elements in an array. Note that in Python, you can also return more than one value, making some of the use cases for pass by reference less important: def foo(x, y): return x**2, y**2 a = 2 b = 3 a, b = foo(a, b) # a == 4; b == 9 When you return values like that, they are being returned as a Tuple which is in turn unpacked. edit: Another way to think about this is that, while you can't explicitly pass variables by reference in Python, you can modify the properties of objects that were passed in. In my example (and others) you can modify members of the list that was passed in. You would not, however, be able to reassign the passed in variable entirely. For instance, see the following two pieces of code look like they might do something similar, but end up with different results: def clear_a(x): x = [] def clear_b(x): while x: x.pop() z = [1,2,3] clear_a(z) # z will not be changed clear_b(z) # z will be emptied


Instruction:
when installing pyaudio, pip cannot find portaudio.h in /usr/local/include

I'm using mac osx 10.10 As the PyAudio Homepage said, I install the PyAudio using brew install portaudio pip install pyaudio the installation of portaudio seems successful, I can find headers and libs in /usr/local/include and /usr/local/lib but when I try to install pyaudio, it gives me an error that src/_portaudiomodule.c:29:10: fatal error: 'portaudio.h' file not found #include "portaudio.h" ^ 1 error generated. error: command 'cc' failed with exit status 1 actually it is in /usr/local/include why can't it find the file? some answers to similar questions are not working for me(like using virtualenv, or compile it manually), and I want to find a simple way to solve this.

Output: Since pyAudio has portAudio as a dependency, you first have to install portaudio. #for Mac brew install portaudio Then try: pip install pyAudio. If the problem persists after installing portAudio, you can specify the directory path where the compiler will be able to find the source programs (e.g: portaudio.h). Since the headers should be in the /usr/local/include directory: pip install --global-option='build_ext' --global-option='-I/usr/local/include' --global-option='-L/usr/local/lib' pyaudio


Instruction:
Is there a way to get a list of column names in sqlite?

I want to get a list of column names from a table in a database. Using pragma I get a list of tuples with a lot of unneeded information. Is there a way to get only the column names? So I might end up with something like this: [Column1, Column2, Column3, Column4] The reason why I absolutely need this list is because I want to search for a column name in the list and get the index because the index is used in a lot of my code. Is there a way of getting a list like this? Thanks

Output: You can use sqlite3 and pep-249 import sqlite3 connection = sqlite3.connect('~/foo.sqlite') cursor = connection.execute('select * from bar') cursor.description is a sequence of 7-item sequences whose first element is the column name: names = list(map(lambda x: x[0], cursor.description)) Alternatively you could use a list comprehension: names = [description[0] for description in cursor.description]


Instruction:
Base 62 conversion

How would you convert an integer to base 62 (like hexadecimal, but with these digits: 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ)? I have been trying to find a good Python library for it, but they all seems to be occupied with converting strings. The Python base64 module only accepts strings and turns a single digit into four characters. I was looking for something akin to what URL shorteners use.

Output: There is no standard module for this, but I have written my own functions to achieve that. BASE62 = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" def encode(num, alphabet): """Encode a positive number into Base X and return the string. Arguments: - `num`: The number to encode - `alphabet`: The alphabet to use for encoding """ if num == 0: return alphabet[0] arr = [] arr_append = arr.append # Extract bound-method for faster access. _divmod = divmod # Access to locals is faster. base = len(alphabet) while num: num, rem = _divmod(num, base) arr_append(alphabet[rem]) arr.reverse() return ''.join(arr) def decode(string, alphabet=BASE62): """Decode a Base X encoded string into the number Arguments: - `string`: The encoded string - `alphabet`: The alphabet to use for decoding """ base = len(alphabet) strlen = len(string) num = 0 idx = 0 for char in string: power = (strlen - (idx + 1)) num += alphabet.index(char) * (base ** power) idx += 1 return num Notice the fact that you can give it any alphabet to use for encoding and decoding. If you leave the alphabet argument out, you are going to get the 62 character alphabet defined on the first line of code, and hence encoding/decoding to/from 62 base. PS - For URL shorteners, I have found that it's better to leave out a few confusing characters like 0Ol1oI etc. Thus I use this alphabet for my URL shortening needs - "23456789abcdefghijkmnpqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ"


Instruction:
Set value for particular cell in pandas DataFrame with iloc

I have a question similar to this and this. The difference is that I have to select row by position, as I do not know the index. I want to do something like df.iloc[0, 'COL_NAME'] = x, but iloc does not allow this kind of access. If I do df.iloc[0]['COL_NAME'] = x the warning about chained indexing appears.

Output: For mixed position and index, use .ix. BUT you need to make sure that your index is not of integer, otherwise it will cause confusions. df.ix[0, 'COL_NAME'] = x Update: Alternatively, try df.iloc[0, df.columns.get_loc('COL_NAME')] = x Example: import pandas as pd import numpy as np # your data # ======================== np.random.seed(0) df = pd.DataFrame(np.random.randn(10, 2), columns=['col1', 'col2'], index=np.random.randint(1,100,10)).sort_index() print(df) col1 col2 10 1.7641 0.4002 24 0.1440 1.4543 29 0.3131 -0.8541 32 0.9501 -0.1514 33 1.8676 -0.9773 36 0.7610 0.1217 56 1.4941 -0.2052 58 0.9787 2.2409 75 -0.1032 0.4106 76 0.4439 0.3337 # .iloc with get_loc # =================================== df.iloc[0, df.columns.get_loc('col2')] = 100 df col1 col2 10 1.7641 100.0000 24 0.1440 1.4543 29 0.3131 -0.8541 32 0.9501 -0.1514 33 1.8676 -0.9773 36 0.7610 0.1217 56 1.4941 -0.2052 58 0.9787 2.2409 75 -0.1032 0.4106 76 0.4439 0.3337


Instruction:
Set value on an entire column of a pandas dataframe

I'm trying to set the entire column of a dataframe to a specific value. In [1]: df Out [1]: issueid industry 0 001 xxx 1 002 xxx 2 003 xxx 3 004 xxx 4 005 xxx From what I've seen, loc is the best practice when replacing values in a dataframe (or isn't it?): In [2]: df.loc[:,'industry'] = 'yyy' However, I still received this much talked-about warning message: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_index,col_indexer] = value instead If I do In [3]: df['industry'] = 'yyy' I got the same warning message. Any ideas? Working with Python 3.5.2 and pandas 0.18.1. EDIT Jan 2023: Given the volume of visits on this question, it's worth stating that my original question was really more about dataframe copy-versus-slice than "setting value to an entire column". On copy-versus-slice: My current understanding is that, in general, if you want to modify a subset of a dataframe after slicing, you should create the subset by .copy(). If you only want a view of the slice, no copy() needed. On setting value to an entire column: simply do df[col_name] = col_value

Output: Python can do unexpected things when new objects are defined from existing ones. You stated in a comment above that your dataframe is defined along the lines of df = df_all.loc[df_all['issueid']==specific_id,:]. In this case, df is really just a stand-in for the rows stored in the df_all object: a new object is NOT created in memory. To avoid these issues altogether, I often have to remind myself to use the copy module, which explicitly forces objects to be copied in memory so that methods called on the new objects are not applied to the source object. I had the same problem as you, and avoided it using the deepcopy function. In your case, this should get rid of the warning message: from copy import deepcopy df = deepcopy(df_all.loc[df_all['issueid']==specific_id,:]) df['industry'] = 'yyy' EDIT: Also see David M.'s excellent comment below! df = df_all.loc[df_all['issueid']==specific_id,:].copy() df['industry'] = 'yyy'


Instruction:
Does python have an equivalent to Java Class.forName()?

I have the need to take a string argument and create an object of the class named in that string in Python. In Java, I would use Class.forName().newInstance(). Is there an equivalent in Python? Thanks for the responses. To answer those who want to know what I'm doing: I want to use a command line argument as the class name, and instantiate it. I'm actually programming in Jython and instantiating Java classes, hence the Java-ness of the question. getattr() works great. Thanks much.

Output: Reflection in python is a lot easier and far more flexible than it is in Java. I recommend reading this tutorial (on archive.org) There's no direct function (that I know of) which takes a fully qualified class name and returns the class, however you have all the pieces needed to build that, and you can connect them together. One bit of advice though: don't try to program in Java style when you're in python. If you can explain what is it that you're trying to do, maybe we can help you find a more pythonic way of doing it. Here's a function that does what you want: def get_class( kls ): parts = kls.split('.') module = ".".join(parts[:-1]) m = __import__( module ) for comp in parts[1:]: m = getattr(m, comp) return m You can use the return value of this function as if it were the class itself. Here's a usage example: >>> D = get_class("datetime.datetime") >>> D <type 'datetime.datetime'> >>> D.now() datetime.datetime(2009, 1, 17, 2, 15, 58, 883000) >>> a = D( 2010, 4, 22 ) >>> a datetime.datetime(2010, 4, 22, 0, 0) >>> How does that work? We're using __import__ to import the module that holds the class, which required that we first extract the module name from the fully qualified name. Then we import the module: m = __import__( module ) In this case, m will only refer to the top level module, For example, if your class lives in foo.baz module, then m will be the module foo We can easily obtain a reference to foo.baz using getattr( m, 'baz' ) To get from the top level module to the class, have to recursively use gettatr on the parts of the class name Say for example, if you class name is foo.baz.bar.Model then we do this: m = __import__( "foo.baz.bar" ) #m is package foo m = getattr( m, "baz" ) #m is package baz m = getattr( m, "bar" ) #m is module bar m = getattr( m, "Model" ) #m is class Model This is what's happening in this loop: for comp in parts[1:]: m = getattr(m, comp) At the end of the loop, m will be a reference to the class. This means that m is actually the class itslef, you can do for instance: a = m() #instantiate a new instance of the class b = m( arg1, arg2 ) # pass arguments to the constructor


Instruction:
Looping from 1 to infinity in Python

In C, I would do this: int i; for (i = 0;; i++) if (thereIsAReasonToBreak(i)) break; How can I achieve something similar in Python?

Output: Using itertools.count: import itertools for i in itertools.count(start=1): if there_is_a_reason_to_break(i): break In Python 2, range() and xrange() were limited to sys.maxsize. In Python 3 range() can go much higher, though not to infinity: import sys for i in range(sys.maxsize**10): # you could go even higher if you really want if there_is_a_reason_to_break(i): break So it's probably best to use count().


Instruction:
How to force integer tick labels

My python script uses matplotlib to plot a 2D "heat map" of an x, y, z dataset. My x- and y-values represent amino acid residues in a protein and can therefore only be integers. When I zoom into the plot, it looks like this: As I said, float values on the x-y axes do not make sense with my data and I therefore want it to look like this: Any ideas how to achieve this? This is the code that generates the plot: def plotDistanceMap(self): # Read on x,y,z x = self.currentGraph['xData'] y = self.currentGraph['yData'] X, Y = numpy.meshgrid(x, y) Z = self.currentGraph['zData'] # Define colormap cmap = colors.ListedColormap(['blue', 'green', 'orange', 'red']) cmap.set_under('white') cmap.set_over('white') bounds = [1,15,50,80,100] norm = colors.BoundaryNorm(bounds, cmap.N) # Draw surface plot img = self.axes.pcolor(X, Y, Z, cmap=cmap, norm=norm) self.axes.set_xlim(x.min(), x.max()) self.axes.set_ylim(y.min(), y.max()) self.axes.set_xlabel(self.currentGraph['xTitle']) self.axes.set_ylabel(self.currentGraph['yTitle']) # Cosmetics #matplotlib.rcParams.update({'font.size': 12}) xminorLocator = MultipleLocator(10) yminorLocator = MultipleLocator(10) self.axes.xaxis.set_minor_locator(xminorLocator) self.axes.yaxis.set_minor_locator(yminorLocator) self.axes.tick_params(direction='out', length=6, width=1) self.axes.tick_params(which='minor', direction='out', length=3, width=1) self.axes.xaxis.labelpad = 15 self.axes.yaxis.labelpad = 15 # Draw colorbar colorbar = self.figure.colorbar(img, boundaries = [0,1,15,50,80,100], spacing = 'proportional', ticks = [15,50,80,100], extend = 'both') colorbar.ax.set_xlabel('Angstrom') colorbar.ax.xaxis.set_label_position('top') colorbar.ax.xaxis.labelpad = 20 self.figure.tight_layout() self.canvas.draw()

Output: This should be simpler: (from https://scivision.co/matplotlib-force-integer-labeling-of-axis/) import matplotlib.pyplot as plt from matplotlib.ticker import MaxNLocator #... ax = plt.figure().gca() #... ax.xaxis.set_major_locator(MaxNLocator(integer=True)) Read the official docs: https://matplotlib.org/stable/api/ticker_api.html#matplotlib.ticker.MaxNLocator


Instruction:
Flask sqlalchemy many-to-many insert data

I am trying to make a many to many relation here in Flask-SQLAlchemy, but it seems that I don't know how to fill the "many to many identifier database". Could you please help me understand what I am doing wrong and how it is supposed to look? class User(db.Model): __tablename__ = 'users' user_id = db.Column(db.Integer, primary_key=True) user_fistName = db.Column(db.String(64)) user_lastName = db.Column(db.String(64)) user_email = db.Column(db.String(128), unique=True) class Class(db.Model): __tablename__ = 'classes' class_id = db.Column(db.Integer, primary_key=True) class_name = db.Column(db.String(128), unique=True) and then my identifier database: student_identifier = db.Table('student_identifier', db.Column('class_id', db.Integer, db.ForeignKey('classes.class_id')), db.Column('user_id', db.Integer, db.ForeignKey('users.user_id')) ) so far it looks like this when I try to insert the data into the database. # User user1 = User( user_fistName='John', user_lastName='Doe', user_email='john@doe.es') user2 = User( user_fistName='Jack', user_lastName='Doe', user_email='jack@doe.es') user3 = User( user_fistName='Jane', user_lastName='Doe', user_email='jane@doe.es') db.session.add_all([user1, user2, user3]) db.session.commit() # Class cl1 = Class(class_name='0A') cl2 = Class(class_name='0B') cl3 = Class(class_name='0C') cl4 = Class(class_name='Math') cl5 = Class(class_name='Spanish') db.session.add_all([cl1, cl2, cl3, cl4, cl5]) db.session.commit() Now my problem is, how do I add to the many to many database, since I really can't create a 'student_identifier' object? If I could it could perhaps have looked like this: # Student Identifier sti1 = StiClass(class_id=cl1.class_id, class_name=user1.user_id) sti2 = StiClass(class_id=cl3.class_id, class_name=user1.user_id) sti3 = StiClass(class_id=cl4.class_id, class_name=user1.user_id) sti4 = StiClass(class_id=cl2.class_id, class_name=user2.user_id) db.session.add_all([sti1, sti2, sti3, sti4]) db.session.commit() How I am supposed to insert into a many to many table with ORM?

Output: You don't need to add anything directly to your association table, SQLAlchemy will do that. This is more or less from SQLAlchemy documentations: association_table = db.Table('association', db.Model.metadata, db.Column('left_id', db.Integer, db.ForeignKey('left.id')), db.Column('right_id', db.Integer, db.ForeignKey('right.id')) ) class Parent(db.Model): __tablename__ = 'left' id = db.Column(db.Integer, primary_key=True) children = db.relationship("Child", secondary=association_table) class Child(db.Model): __tablename__ = 'right' id = db.Column(db.Integer, primary_key=True) p = Parent() c = Child() p.children.append(c) db.session.add(p) db.session.commit() Therefore your sample would be like this: student_identifier = db.Table('student_identifier', db.Column('class_id', db.Integer, db.ForeignKey('classes.class_id')), db.Column('user_id', db.Integer, db.ForeignKey('students.user_id')) ) class Student(db.Model): __tablename__ = 'students' user_id = db.Column(db.Integer, primary_key=True) user_fistName = db.Column(db.String(64)) user_lastName = db.Column(db.String(64)) user_email = db.Column(db.String(128), unique=True) class Class(db.Model): __tablename__ = 'classes' class_id = db.Column(db.Integer, primary_key=True) class_name = db.Column(db.String(128), unique=True) students = db.relationship("Student", secondary=student_identifier) s = Student() c = Class() c.students.append(s) db.session.add(c) db.session.commit()


Instruction:
How to implement band-pass Butterworth filter with Scipy.signal.butter

UPDATE: I found a Scipy Recipe based in this question! So, for anyone interested, go straight to: Contents » Signal processing » Butterworth Bandpass I'm having a hard time to achieve what seemed initially a simple task of implementing a Butterworth band-pass filter for 1-D numpy array (time-series). The parameters I have to include are the sample_rate, cutoff frequencies IN HERTZ and possibly order (other parameters, like attenuation, natural frequency, etc. are more obscure to me, so any "default" value would do). What I have now is this, which seems to work as a high-pass filter but I'm no way sure if I'm doing it right: def butter_highpass(interval, sampling_rate, cutoff, order=5): nyq = sampling_rate * 0.5 stopfreq = float(cutoff) cornerfreq = 0.4 * stopfreq # (?) ws = cornerfreq/nyq wp = stopfreq/nyq # for bandpass: # wp = [0.2, 0.5], ws = [0.1, 0.6] N, wn = scipy.signal.buttord(wp, ws, 3, 16) # (?) # for hardcoded order: # N = order b, a = scipy.signal.butter(N, wn, btype='high') # should 'high' be here for bandpass? sf = scipy.signal.lfilter(b, a, interval) return sf The docs and examples are confusing and obscure, but I'd like to implement the form presented in the commend marked as "for bandpass". The question marks in the comments show where I just copy-pasted some example without understanding what is happening. I am no electrical engineering or scientist, just a medical equipment designer needing to perform some rather straightforward bandpass filtering on EMG signals.

Output: You could skip the use of buttord, and instead just pick an order for the filter and see if it meets your filtering criterion. To generate the filter coefficients for a bandpass filter, give butter() the filter order, the cutoff frequencies Wn=[lowcut, highcut], the sampling rate fs (expressed in the same units as the cutoff frequencies) and the band type btype="band". Here's a script that defines a couple convenience functions for working with a Butterworth bandpass filter. When run as a script, it makes two plots. One shows the frequency response at several filter orders for the same sampling rate and cutoff frequencies. The other plot demonstrates the effect of the filter (with order=6) on a sample time series. from scipy.signal import butter, lfilter def butter_bandpass(lowcut, highcut, fs, order=5): return butter(order, [lowcut, highcut], fs=fs, btype='band') def butter_bandpass_filter(data, lowcut, highcut, fs, order=5): b, a = butter_bandpass(lowcut, highcut, fs, order=order) y = lfilter(b, a, data) return y if __name__ == "__main__": import numpy as np import matplotlib.pyplot as plt from scipy.signal import freqz # Sample rate and desired cutoff frequencies (in Hz). fs = 5000.0 lowcut = 500.0 highcut = 1250.0 # Plot the frequency response for a few different orders. plt.figure(1) plt.clf() for order in [3, 6, 9]: b, a = butter_bandpass(lowcut, highcut, fs, order=order) w, h = freqz(b, a, fs=fs, worN=2000) plt.plot(w, abs(h), label="order = %d" % order) plt.plot([0, 0.5 * fs], [np.sqrt(0.5), np.sqrt(0.5)], '--', label='sqrt(0.5)') plt.xlabel('Frequency (Hz)') plt.ylabel('Gain') plt.grid(True) plt.legend(loc='best') # Filter a noisy signal. T = 0.05 nsamples = T * fs t = np.arange(0, nsamples) / fs a = 0.02 f0 = 600.0 x = 0.1 * np.sin(2 * np.pi * 1.2 * np.sqrt(t)) x += 0.01 * np.cos(2 * np.pi * 312 * t + 0.1) x += a * np.cos(2 * np.pi * f0 * t + .11) x += 0.03 * np.cos(2 * np.pi * 2000 * t) plt.figure(2) plt.clf() plt.plot(t, x, label='Noisy signal') y = butter_bandpass_filter(x, lowcut, highcut, fs, order=6) plt.plot(t, y, label='Filtered signal (%g Hz)' % f0) plt.xlabel('time (seconds)') plt.hlines([-a, a], 0, T, linestyles='--') plt.grid(True) plt.axis('tight') plt.legend(loc='upper left') plt.show() Here are the plots that are generated by this script:


Instruction:
Multiple datasets on the same scatter plot

I want to plot multiple data sets on the same scatter plot: cases = scatter(x[:4], y[:4], s=10, c='b', marker="s") controls = scatter(x[4:], y[4:], s=10, c='r', marker="o") show() The above only shows the most recent scatter() I've also tried: plt = subplot(111) plt.scatter(x[:4], y[:4], s=10, c='b', marker="s") plt.scatter(x[4:], y[4:], s=10, c='r', marker="o") show()

Output: You need a reference to an Axes object to keep drawing on the same subplot. import matplotlib.pyplot as plt x = range(100) y = range(100,200) fig = plt.figure() ax1 = fig.add_subplot(111) ax1.scatter(x[:4], y[:4], s=10, c='b', marker="s", label='first') ax1.scatter(x[40:],y[40:], s=10, c='r', marker="o", label='second') plt.legend(loc='upper left') plt.show()


Instruction:
Dataclasses and property decorator

I've been reading up on Python 3.7's dataclass as an alternative to namedtuples (what I typically use when having to group data in a structure). I was wondering if dataclass is compatible with the property decorator to define getter and setter functions for the data elements of the dataclass. If so, is this described somewhere? Or are there examples available?

Output: It sure does work: from dataclasses import dataclass @dataclass class Test: _name: str="schbell" @property def name(self) -> str: return self._name @name.setter def name(self, v: str) -> None: self._name = v t = Test() print(t.name) # schbell t.name = "flirp" print(t.name) # flirp print(t) # Test(_name='flirp') In fact, why should it not? In the end, what you get is just a good old class, derived from type: print(type(t)) # <class '__main__.Test'> print(type(Test)) # <class 'type'> Maybe that's why properties are nowhere mentioned specifically. However, the PEP-557's Abstract mentions the general usability of well-known Python class features: Because Data Classes use normal class definition syntax, you are free to use inheritance, metaclasses, docstrings, user-defined methods, class factories, and other Python class features.


Instruction:
Strange result when removing item from a list while iterating over it in Python

I've got this code: numbers = list(range(1, 50)) for i in numbers: if i < 20: numbers.remove(i) print(numbers) This code should go through all the elements of the list and remove those elements that are less than 20. But, the result I'm getting is: [2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49] As you can see, the code removed all even numbers less than 20, but left the odd ones. Looks like I'm doing something wrong with the remove.

Output: You're modifying the list while you iterate over it. That means that the first time through the loop, i == 1, so 1 is removed from the list. Then the for loop goes to the second item in the list, which is not 2, but 3! Then that's removed from the list, and then the for loop goes on to the third item in the list, which is now 5. And so on. Perhaps it's easier to visualize like so, with a ^ pointing to the value of i: [1, 2, 3, 4, 5, 6...] ^ That's the state of the list initially; then 1 is removed and the loop goes to the second item in the list: [2, 3, 4, 5, 6...] ^ [2, 4, 5, 6...] ^ And so on. There's no good way to alter a list's length while iterating over it. The best you can do is something like this: numbers = [n for n in numbers if n >= 20] or this, for in-place alteration (the thing in parens is a generator expression, which is implicitly converted into a tuple before slice-assignment): numbers[:] = (n for n in numbers if n >= 20) If you want to perform an operation on n before removing it, one trick you could try is this: for i, n in enumerate(numbers): if n < 20: print("do something") numbers[i] = None numbers = [n for n in numbers if n is not None]


Instruction:
See if two files have the same content in python

Possible Duplicates: Finding duplicate files and removing them. In Python, is there a concise way of comparing whether the contents of two text files are the same? What is the easiest way to see if two files are the same content-wise in Python. One thing I can do is md5 each file and compare. Is there a better way?

Output: Yes, I think hashing the file would be the best way if you have to compare several files and store hashes for later comparison. As hash can clash, a byte-by-byte comparison may be done depending on the use case. Generally byte-by-byte comparison would be sufficient and efficient, which filecmp module already does + other things too. See http://docs.python.org/library/filecmp.html e.g. >>> import filecmp >>> filecmp.cmp('file1.txt', 'file1.txt') True >>> filecmp.cmp('file1.txt', 'file2.txt') False Note that by default, filecmp does not compare the contents of the files, to do so, add a third parameter shallow=False. Speed consideration: Usually if only two files have to be compared, hashing them and comparing them would be slower instead of simple byte-by-byte comparison if done efficiently. e.g. code below tries to time hash vs byte-by-byte Disclaimer: this is not the best way of timing or comparing two algo. and there is need for improvements but it does give rough idea. If you think it should be improved do tell me I will change it. import random import string import hashlib import time def getRandText(N): return "".join([random.choice(string.printable) for i in xrange(N)]) N=1000000 randText1 = getRandText(N) randText2 = getRandText(N) def cmpHash(text1, text2): hash1 = hashlib.md5() hash1.update(text1) hash1 = hash1.hexdigest() hash2 = hashlib.md5() hash2.update(text2) hash2 = hash2.hexdigest() return hash1 == hash2 def cmpByteByByte(text1, text2): return text1 == text2 for cmpFunc in (cmpHash, cmpByteByByte): st = time.time() for i in range(10): cmpFunc(randText1, randText2) print cmpFunc.func_name,time.time()-st and the output is cmpHash 0.234999895096 cmpByteByByte 0.0


Instruction:
How to forget previous plots - how can I flush/refresh?

How do you get matplotlib.pyplot to "forget" previous plots I am trying to plot multiple time using matplotlib.pyplot The code looks like this: def plottest(): import numpy as np import matplotlib.pyplot as plt a=np.random.rand(10,) b=np.random.rand(10,) c=np.random.rand(10,) plt.plot(a,label='a') plt.plot(b,label='b') plt.plot(c,label='c') plt.legend(loc='upper left') plt.ylabel('mag') plt.xlabel('element)') plt.show() e=np.random.rand(10,) f=np.random.rand(10,) g=np.random.rand(10,) plt.plot(e,label='e') plt.plot(f,label='f') plt.plot(g,label='g') plt.legend(loc='upper left') plt.ylabel('mag') plt.xlabel('element)') plt.show() Unfortunately I keep getting the same plot (actually from some other code which I ran and completed a while ago) no matter what I do. Similar code has worked previously for me. I have looked at these questions: How to "clean the slate"? Matplotlib pyplot show() doesn't work once closed (python) matplotlib pyplot show() .. blocking or not? and tried using plt.show(), plt.clf() and plt.close to no avail. Any ideas?

Output: I would rather use plt.clf() after every plt.show() to just clear the current figure instead of closing and reopening it, keeping the window size and giving you a better performance and much better memory usage. Similarly, you could do plt.cla() to just clear the current axes. To clear a specific axes, useful when you have multiple axes within one figure, you could do for example: fig, axes = plt.subplots(nrows=2, ncols=2) axes[0, 1].clear()


Instruction:
Check if string is upper, lower, or mixed case in Python

I want to classify a list of string in Python depending on whether they are upper case, lower case, or mixed case How can I do this?

Output: There are a number of "is methods" on strings. islower() and isupper() should meet your needs: >>> 'hello'.islower() True >>> [m for m in dir(str) if m.startswith('is')] ['isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper'] Here's an example of how to use those methods to classify a list of strings: >>> words = ['The', 'quick', 'BROWN', 'Fox', 'jumped', 'OVER', 'the', 'Lazy', 'DOG'] >>> [word for word in words if word.islower()] ['quick', 'jumped', 'the'] >>> [word for word in words if word.isupper()] ['BROWN', 'OVER', 'DOG'] >>> [word for word in words if not word.islower() and not word.isupper()] ['The', 'Fox', 'Lazy']


Instruction:
Overloaded functions in Python

Is it possible to have overloaded functions in Python? In C# I would do something like: void myfunction (int first, string second) { // Some code } void myfunction (int first, string second, float third) { // Some different code } And then when I call the function it would differentiate between the two based on the number of arguments. Is it possible to do something similar in Python?

Output: EDIT For the new single dispatch generic functions in Python 3.4, see http://www.python.org/dev/peps/pep-0443/ You generally don't need to overload functions in Python. Python is dynamically typed, and supports optional arguments to functions. def myfunction(first, second, third = None): if third is None: #just use first and second else: #use all three myfunction(1, 2) # third will be None, so enter the 'if' clause myfunction(3, 4, 5) # third isn't None, it's 5, so enter the 'else' clause


Instruction:
How do you add additional files to a wheel?

How do control what files are included in a wheel? It appears MANIFEST.in isn't used by python setup.py bdist_wheel. UPDATE: I was wrong about the difference between installing from a source tarball vs a wheel. The source distribution includes files specified in MANIFEST.in, but the installed package only has python files. Steps are needed to identify additional files that should be installed, whether the install is via source distribution, egg, or wheel. Namely, package_data is needed for additional package files, and data_files for files outside your package like command line scripts or system config files. Original Question I have a project where I've been using python setup.py sdist to build my package, MANIFEST.in to control the files included and excluded, and pyroma and check-manifest to confirm my settings. I recently converted it to dual Python 2 / 3 code, and added a setup.cfg with [bdist_wheel] universal = 1 I can build a wheel with python setup.py bdist_wheel, and it appears to be a universal wheel as desired. However, it doesn't include all of the files specified in MANIFEST.in. What gets installed? I dug deeper, and now know more about packaging and wheel. Here's what I learned: I upload two package files to the multigtfs project on PyPi: multigtfs-0.4.2.tar.gz - the source tar ball, which includes all the files in MANIFEST.in. multigtfs-0.4.2-py2.py3-none-any.whl - The binary distribution in question. I created two new virtual environments, both with Python 2.7.5, and installed each package (pip install multigtfs-0.4.2.tar.gz). The two environments are almost identical. They have different .pyc files, which are the "compiled" Python files. There are log files which record the different paths on disk. The install from the source tar ball includes a folder multigtfs-0.4.2-py27.egg-info, detailing the installation, and the wheel install has a multigtfs-0.4.2.dist-info folder, with the details of that process. However, from the point of view of code using the multigtfs project, there is no difference between the two installation methods. Explicitly, neither has the .zip files used by my test, so the test suite will fail: $ django-admin startproject demo $ cd demo $ pip install psycopg2 # DB driver for PostGIS project $ createdb demo # Create PostgreSQL database $ psql -d demo -c "CREATE EXTENSION postgis" # Make it a PostGIS database $ vi demo/settings.py # Add multigtfs to INSTALLED_APPS, # Update DATABASE to set ENGINE to django.contrib.gis.db.backends.postgis # Update DATABASE to set NAME to test $ ./manage.py test multigtfs.tests # Run the tests ... IOError: [Errno 2] No such file or directory: u'/Users/john/.virtualenvs/test/lib/python2.7/site-packages/multigtfs/tests/fixtures/test3.zip' Specifying additional files Using the suggestions from the answers, I added some additional directives to setup.py: from __future__ import unicode_literals # setup.py now requires some funky binary strings ... setup( name='multigtfs', packages=find_packages(), package_data={b'multigtfs': ['test/fixtures/*.zip']}, include_package_data=True, ... ) This installs the zip files (as well as the README) to the folder, and tests now run correctly. Thanks for the suggestions!

Output: Have you tried using package_data in your setup.py? MANIFEST.in seems targetted for python versions <= 2.6, I'm not sure if higher versions even look at it. After exploring https://github.com/pypa/sampleproject, their MANIFEST.in says: # If using Python 2.6 or less, then have to include package data, even though # it's already declared in setup.py include sample/*.dat which seems to imply this method is outdated. Meanwhile, in setup.py they declare: setup( name='sample', ... # If there are data files included in your packages that need to be # installed, specify them here. If using Python 2.6 or less, then these # have to be included in MANIFEST.in as well. include_package_data=True, package_data={ 'sample': ['package_data.dat'], }, ... ) (I'm not sure why they chose a wildcard in MANIFEST.in and a filename in setup.py. They refer to the same file) Which, along with being simpler, again seems to imply that the package_data route is superior to the MANIFEST.in method. Well, unless you have to support 2.6 that is, in which case my prayers go out to you.


Instruction:
How to add/subtract time (hours, minutes, etc.) from a Pandas DataFrame.Index whos objects are of type datetime.time?

I have a DataFrame whose index values are of type datetime.time. There is no method in DataFrame.Index to shift the time. datetime.time has replace but that will only work on individual items. Here's an example: In[526]: dfa.index[:5] Out[526]: Index([21:12:19, 21:12:20, 21:12:21, 21:12:21, 21:12:22], dtype='object') In[527]: type(dfa.index[0]) Out[527]: datetime.time

Output: Liam's link looks great, but also check out pandas.Timedelta - looks like it plays nicely with NumPy's and Python's time deltas. https://pandas.pydata.org/pandas-docs/stable/timedeltas.html pd.date_range('2014-01-01', periods=10) + pd.Timedelta(days=1)


Instruction:
Python: fastest way to create a list of n lists

So I was wondering how to best create a list of blank lists: [[],[],[]...] Because of how Python works with lists in memory, this doesn't work: [[]]*n This does create [[],[],...] but each element is the same list: d = [[]]*n d[0].append(1) #[[1],[1],...] Something like a list comprehension works: d = [[] for x in xrange(0,n)] But this uses the Python VM for looping. Is there any way to use an implied loop (taking advantage of it being written in C)? d = [] map(lambda n: d.append([]),xrange(0,10)) This is actually slower. :(

Output: The probably only way which is marginally faster than d = [[] for x in xrange(n)] is from itertools import repeat d = [[] for i in repeat(None, n)] It does not have to create a new int object in every iteration and is about 15 % faster on my machine. Edit: Using NumPy, you can avoid the Python loop using d = numpy.empty((n, 0)).tolist() but this is actually 2.5 times slower than the list comprehension.


Instruction:
Why is f'{{{74}}}' the same as f'{{74}}' with f-Strings?

f-Strings are available from Python 3.6 and are very useful for formatting strings: >>> n='you' >>> f'hello {n}, how are you?' 'hello you, how are you?' Reading more about them in Python 3's f-Strings: An Improved String Formatting Syntax (Guide). I found an interesting pattern: Note that using triple braces will result in there being only single braces in your string: >>> f"{{{74}}}" '{74}' However, you can get more braces to show if you use more than triple braces: >>> f"{{{{74}}}}" '{{74}}' And this is exactly the case: >>> f'{74}' '74' >>> f'{{74}}' '{74}' Now if we pass from two { to three, the result is the same: >>> f'{{{74}}}' '{74}' # same as f'{{74}}' ! So we need up to 4! ({{{{) to get two braces as an output: >>> f'{{{{74}}}}' '{{74}}' Why is this? What happens with two braces to have Python require an extra one from that moment on?

Output: Double braces escape the braces, so that no interpolation happens: {{ ➝ {, and }} ➝ }. And 74 remains an unchanged string, '74'. With triple braces, the outer double braces are escaped, same as above. The inner braces, on the other hand, lead to regular string interpolation of the value 74. That is, the string f'{{{74}}}' is equivalent to f'{{ {74} }}', but without spaces (or, equivalently, to '{' + f'{74}' + '}'). You can see the difference when replacing the numeric constant by a variable: In [1]: x = 74 In [2]: f'{{x}}' Out[2]: '{x}' In [3]: f'{{{x}}}' Out[3]: '{74}'


Instruction:
Proper use of mutexes in Python

I am starting with multi-threads in python (or at least it is possible that my script creates multiple threads). would this algorithm be the right usage of a Mutex? I haven't tested this code yet and it probably won't even work. I just want processData to run in a thread (one at time) and the main while loop to keep running, even if there is a thread in queue. from threading import Thread from win32event import CreateMutex mutex = CreateMutex(None, False, "My Crazy Mutex") while(1) t = Thread(target=self.processData, args=(some_data,)) t.start() mutex.lock() def processData(self, data) while(1) if mutex.test() == False: do some stuff break Edit: re-reading my code I can see that it is grossly wrong. but hey, that's why I am here asking for help.

Output: I don't know why you're using the Window's Mutex instead of Python's. Using the Python methods, this is pretty simple: from threading import Thread, Lock mutex = Lock() def processData(data): with mutex: print('Do some stuff') while True: t = Thread(target = processData, args = (some_data,)) t.start() But note, because of the architecture of CPython (namely the Global Interpreter Lock) you'll effectively only have one thread running at a time anyway--this is fine if a number of them are I/O bound, although you'll want to release the lock as much as possible so the I/O bound thread doesn't block other threads from running. An alternative, for Python 2.6 and later, is to use Python's multiprocessing package. It mirrors the threading package, but will create entirely new processes which can run simultaneously. It's trivial to update your example: from multiprocessing import Process, Lock mutex = Lock() def processData(data): with mutex: print('Do some stuff') if __name__ == '__main__': while True: p = Process(target = processData, args = (some_data,)) p.start()


Instruction:
Assign variable in while loop condition in Python?

I just came across this piece of code while 1: line = data.readline() if not line: break #... and thought, there must be a better way to do this, than using an infinite loop with break. So I tried: while line = data.readline(): #... and, obviously, got an error. Is there any way to avoid using a break in that situation? Edit: Ideally, you'd want to avoid saying readline twice... IMHO, repeating is even worse than just a break, especially if the statement is complex.

Output: Starting Python 3.8, and the introduction of assignment expressions (PEP 572) (:= operator), it's now possible to capture the condition value (data.readline()) of the while loop as a variable (line) in order to re-use it within the body of the loop: while line := data.readline(): do_smthg(line)


Instruction:
How to find out the request.session sessionid and use it as a variable in Django?

I'm aware that you can get session variables using request.session['variable_name'], but there doesn't seem to be a way to grab the session id(key) as a variable in a similar way. Is this documented anywhere? I can't find it.

Output: request.session.session_key Note the key will only exist if there is a session, no key, no session. You can use this to test if a session exists. If you want to create a session, call create.


Instruction:
Decode escaped characters in URL

I have a list containing URLs with escaped characters in them. Those characters have been set by urllib2.urlopen when it recovers the html page: http://www.sample1webpage.com/index.php?title=%E9%A6%96%E9%A1%B5&action=edit http://www.sample1webpage.com/index.php?title=%E9%A6%96%E9%A1%B5&action=history http://www.sample1webpage.com/index.php?title=%E9%A6%96%E9%A1%B5&variant=zh Is there a way to transform them back to their unescaped form in python? P.S.: The URLs are encoded in utf-8

Output: Using urllib package (import urllib) : Python 2.7 From official documentation : urllib.unquote(string) Replace %xx escapes by their single-character equivalent. Example: unquote('/%7Econnolly/') yields '/~connolly/'. Python 3 From official documentation : urllib.parse.unquote(string, encoding='utf-8', errors='replace') […] Example: unquote('/El%20Ni%C3%B1o/') yields '/El Niño/'.


Instruction:
Use tqdm with concurrent.futures?

I have a multithreaded function that I would like a status bar for using tqdm. Is there an easy way to show a status bar with ThreadPoolExecutor? It is the parallelization part that is confusing me. import concurrent.futures def f(x): return f**2 my_iter = range(1000000) def run(f,my_iter): with concurrent.futures.ThreadPoolExecutor() as executor: function = list(executor.map(f, my_iter)) return results run(f, my_iter) # wrap tqdr around this function?

Output: You can wrap tqdm around the executor as the following to track the progress: list(tqdm(executor.map(f, iter), total=len(iter)) Here is your example: import time import concurrent.futures from tqdm import tqdm def f(x): time.sleep(0.001) # to visualize the progress return x**2 def run(f, my_iter): with concurrent.futures.ThreadPoolExecutor() as executor: results = list(tqdm(executor.map(f, my_iter), total=len(my_iter))) return results my_iter = range(100000) run(f, my_iter) And the result is like this: 16%|██▏ | 15707/100000 [00:00<00:02, 31312.54it/s]


Instruction:
How to append rows in a pandas dataframe in a for loop?

I have the following for loop: for i in links: data = urllib2.urlopen(str(i)).read() data = json.loads(data) data = pd.DataFrame(data.items()) data = data.transpose() data.columns = data.iloc[0] data = data.drop(data.index[[0]]) Each dataframe so created has most columns in common with the others but not all of them. Moreover, they all have just one row. What I need to to is to add to the dataframe all the distinct columns and each row from each dataframe produced by the for loop I tried pandas concatenate or similar but nothing seemed to work. Any idea? Thanks.

Output: Suppose your data looks like this: import pandas as pd import numpy as np np.random.seed(2015) df = pd.DataFrame([]) for i in range(5): data = dict(zip(np.random.choice(10, replace=False, size=5), np.random.randint(10, size=5))) data = pd.DataFrame(data.items()) data = data.transpose() data.columns = data.iloc[0] data = data.drop(data.index[[0]]) df = df.append(data) print('{}\n'.format(df)) # 0 0 1 2 3 4 5 6 7 8 9 # 1 6 NaN NaN 8 5 NaN NaN 7 0 NaN # 1 NaN 9 6 NaN 2 NaN 1 NaN NaN 2 # 1 NaN 2 2 1 2 NaN 1 NaN NaN NaN # 1 6 NaN 6 NaN 4 4 0 NaN NaN NaN # 1 NaN 9 NaN 9 NaN 7 1 9 NaN NaN Then it could be replaced with np.random.seed(2015) data = [] for i in range(5): data.append(dict(zip(np.random.choice(10, replace=False, size=5), np.random.randint(10, size=5)))) df = pd.DataFrame(data) print(df) In other words, do not form a new DataFrame for each row. Instead, collect all the data in a list of dicts, and then call df = pd.DataFrame(data) once at the end, outside the loop. Each call to df.append requires allocating space for a new DataFrame with one extra row, copying all the data from the original DataFrame into the new DataFrame, and then copying data into the new row. All that allocation and copying makes calling df.append in a loop very inefficient. The time cost of copying grows quadratically with the number of rows. Not only is the call-DataFrame-once code easier to write, its performance will be much better -- the time cost of copying grows linearly with the number of rows.


Instruction:
How to use `async for` in Python?

I mean what do I get from using async for. Here is the code I write with async for, AIter(10) could be replaced with get_range(). But the code runs like sync not async. import asyncio async def get_range(): for i in range(10): print(f"start {i}") await asyncio.sleep(1) print(f"end {i}") yield i class AIter: def __init__(self, N): self.i = 0 self.N = N def __aiter__(self): return self async def __anext__(self): i = self.i print(f"start {i}") await asyncio.sleep(1) print(f"end {i}") if i >= self.N: raise StopAsyncIteration self.i += 1 return i async def main(): async for p in AIter(10): print(f"finally {p}") if __name__ == "__main__": asyncio.run(main()) The result I excepted should be : start 1 start 2 start 3 ... end 1 end 2 ... finally 1 finally 2 ... However, the real result is: start 0 end 0 finally 0 start 1 end 1 finally 1 start 2 end 2 I know I could get the excepted result by using asyncio.gather or asyncio.wait. But it is hard for me to understand what I got by use async for here instead of simple for. What is the right way to use async for if I want to loop over several Feature object and use them as soon as one is finished. For example: async for f in feature_objects: data = await f with open("file", "w") as fi: fi.write()

Output: But it is hard for me to understand what I got by use async for here instead of simple for. The underlying misunderstanding is expecting async for to automatically parallelize the iteration. It doesn't do that, it simply allows sequential iteration over an async source. For example, you can use async for to iterate over lines coming from a TCP stream, messages from a websocket, or database records from an async DB driver. The iteration being async means that you can run it in parallel with other async tasks (including other such iterations) in the same event loop. Ordinary for is incapable of async iteration, at least not without blocking the thread it's running in. This is because for calls __next__ as a blocking function and doesn't await its result. And you cannot manually await elements obtained by for because for expects __next__ to signal the end of iteration by raising StopIteration. If __next__ is a coroutine, the StopIteration exception won't be visible before awaiting it. This is why async for was introduced, not just in Python, but also in other languages with async/await and generalized for. In other words, while ordinary for foo in bar(): ... desugars to something like: __it = bar().__iter__() while True: try: foo = __it.__next__() # await missing except StopIteration: break ... ...async for foo in bar(): ... desugars to: __ait = bar().__aiter__() while True: try: foo = await __ait.__anext__() # await present except StopAsyncIteration: break ... If you want to run the loop iterations in parallel, you need to start them as parallel coroutines and use asyncio.as_completed or equivalent to retrieve their results as they come: async def x(i): print(f"start {i}") await asyncio.sleep(1) print(f"end {i}") return i # run x(0)..x(10) concurrently and process results as they arrive for f in asyncio.as_completed([x(i) for i in range(10)]): result = await f # ... do something with the result ... If you don't care about reacting to results immediately as they arrive, but you need them all, you can make it even simpler by using asyncio.gather: # run x(0)..x(10) concurrently and process results when all are done results = await asyncio.gather(*[x(i) for i in range(10)])


Instruction:
no module named 'dotenv' python 3.8

I've tried reinstalling both dotenv and python-dotenv but I'm still getting the same error. I do have the .env file in the same directory as this script. #bot.py import os import discord from dotenv import load_dotenv load_dotenv() token=os.getenv('DISCORD_TOKEN') client = discord.Client() @client.event async def on_ready(): print(f'{client.user} has connected to Discord!') client.run(token)

Output: in your installation manager if it's Ubuntu or Debian try: apt install python3-dotenv you can also try sudo pip3 install dotenv to install via pip. Whatever you do remember to include explicitly the missing 3 part. Debian/Ubuntu have separate packages and as of the present time python means python2 and python3 means python3 in their apt repositories. However, when it comes to your locally installed python binary on your system which python binary it defaults to using may vary depending on what /usr/bin/python is symlinked to on your system. Some systems it's symlinked to something like python2.7 and other's it may be something like python3.5. Similar issues exist with locally installed pip. Hence, why using the '3' is important when installing or searching for python packages


Instruction:
Default arguments with *args and **kwargs

In Python 2.x (I use 2.7), which is the proper way to use default arguments with *args and **kwargs? I've found a question on SO related to this topic, but that is for Python 3: Calling a Python function with *args,**kwargs and optional / default arguments There, they say this method works: def func(arg1, arg2, *args, opt_arg='def_val', **kwargs): #... In 2.7, it results in a SyntaxError. Is there any recommended way to define such a function? I got it working this way, but I'd guess there is a nicer solution. def func(arg1, arg2, *args, **kwargs): opt_arg ='def_val' if kwargs.__contains__('opt_arg'): opt_arg = kwargs['opt_arg'] #...

Output: Just put the default arguments before the *args: def foo(a, b=3, *args, **kwargs): Now, b will be explicitly set if you pass it as a keyword argument or the second positional argument. Examples: foo(x) # a=x, b=3, args=(), kwargs={} foo(x, y) # a=x, b=y, args=(), kwargs={} foo(x, b=y) # a=x, b=y, args=(), kwargs={} foo(x, y, z, k) # a=x, b=y, args=(z, k), kwargs={} foo(x, c=y, d=k) # a=x, b=3, args=(), kwargs={'c': y, 'd': k} foo(x, c=y, b=z, d=k) # a=x, b=z, args=(), kwargs={'c': y, 'd': k} Note that, in particular, foo(x, y, b=z) doesn't work because b is assigned by position in that case. This code works in Python 3 too. Putting the default arg after *args in Python 3 makes it a "keyword-only" argument that can only be specified by name, not by position. If you want a keyword-only argument in Python 2, you can use @mgilson's solution.