Instruction:
How to have clusters of stacked bars

So here is how my data set looks like : In [1]: df1=pd.DataFrame(np.random.rand(4,2),index=["A","B","C","D"],columns=["I","J"]) In [2]: df2=pd.DataFrame(np.random.rand(4,2),index=["A","B","C","D"],columns=["I","J"]) In [3]: df1 Out[3]: I J A 0.675616 0.177597 B 0.675693 0.598682 C 0.631376 0.598966 D 0.229858 0.378817 In [4]: df2 Out[4]: I J A 0.939620 0.984616 B 0.314818 0.456252 C 0.630907 0.656341 D 0.020994 0.538303 I want to have stacked bar plot for each dataframe but since they have same index, I'd like to have 2 stacked bars per index. I've tried to plot both on the same axes : In [5]: ax = df1.plot(kind="bar", stacked=True) In [5]: ax2 = df2.plot(kind="bar", stacked=True, ax = ax) But it overlaps. Then I tried to concat the two dataset first : pd.concat(dict(df1 = df1, df2 = df2),axis = 1).plot(kind="bar", stacked=True) but here everything is stacked My best try is : pd.concat(dict(df1 = df1, df2 = df2),axis = 0).plot(kind="bar", stacked=True) Which gives : This is basically what I want, except that I want the bar ordered as (df1,A) (df2,A) (df1,B) (df2,B) etc... I guess there is a trick but I can't found it ! After @bgschiller's answer I got this : Which is almost what I want. I would like the bar to be clustered by index, in order to have something visually clear. Bonus : Having the x-label not redundant, something like : df1 df2 df1 df2 _______ _______ ... A B

Output: I eventually found a trick (edit: see below for using seaborn and longform dataframe): Solution with pandas and matplotlib Here it is with a more complete example : import pandas as pd import matplotlib.cm as cm import numpy as np import matplotlib.pyplot as plt def plot_clustered_stacked(dfall, labels=None, title="multiple stacked bar plot", H="/", **kwargs): """Given a list of dataframes, with identical columns and index, create a clustered stacked bar plot. labels is a list of the names of the dataframe, used for the legend title is a string for the title of the plot H is the hatch used for identification of the different dataframe""" n_df = len(dfall) n_col = len(dfall[0].columns) n_ind = len(dfall[0].index) axe = plt.subplot(111) for df in dfall : # for each data frame axe = df.plot(kind="bar", linewidth=0, stacked=True, ax=axe, legend=False, grid=False, **kwargs) # make bar plots h,l = axe.get_legend_handles_labels() # get the handles we want to modify for i in range(0, n_df * n_col, n_col): # len(h) = n_col * n_df for j, pa in enumerate(h[i:i+n_col]): for rect in pa.patches: # for each index rect.set_x(rect.get_x() + 1 / float(n_df + 1) * i / float(n_col)) rect.set_hatch(H * int(i / n_col)) #edited part rect.set_width(1 / float(n_df + 1)) axe.set_xticks((np.arange(0, 2 * n_ind, 2) + 1 / float(n_df + 1)) / 2.) axe.set_xticklabels(df.index, rotation = 0) axe.set_title(title) # Add invisible data to add another legend n=[] for i in range(n_df): n.append(axe.bar(0, 0, color="gray", hatch=H * i)) l1 = axe.legend(h[:n_col], l[:n_col], loc=[1.01, 0.5]) if labels is not None: l2 = plt.legend(n, labels, loc=[1.01, 0.1]) axe.add_artist(l1) return axe # create fake dataframes df1 = pd.DataFrame(np.random.rand(4, 5), index=["A", "B", "C", "D"], columns=["I", "J", "K", "L", "M"]) df2 = pd.DataFrame(np.random.rand(4, 5), index=["A", "B", "C", "D"], columns=["I", "J", "K", "L", "M"]) df3 = pd.DataFrame(np.random.rand(4, 5), index=["A", "B", "C", "D"], columns=["I", "J", "K", "L", "M"]) # Then, just call : plot_clustered_stacked([df1, df2, df3],["df1", "df2", "df3"]) And it gives that : You can change the colors of the bar by passing a cmap argument: plot_clustered_stacked([df1, df2, df3], ["df1", "df2", "df3"], cmap=plt.cm.viridis) Solution with seaborn: Given the same df1, df2, df3, below, I convert them in a long form: df1["Name"] = "df1" df2["Name"] = "df2" df3["Name"] = "df3" dfall = pd.concat([pd.melt(i.reset_index(), id_vars=["Name", "index"]) # transform in tidy format each df for i in [df1, df2, df3]], ignore_index=True) The problem with seaborn is that it doesn't stack bars natively, so the trick is to plot the cumulative sum of each bar on top of each other: dfall.set_index(["Name", "index", "variable"], inplace=1) dfall["vcs"] = dfall.groupby(level=["Name", "index"]).cumsum() dfall.reset_index(inplace=True) >>> dfall.head(6) Name index variable value vcs 0 df1 A I 0.717286 0.717286 1 df1 B I 0.236867 0.236867 2 df1 C I 0.952557 0.952557 3 df1 D I 0.487995 0.487995 4 df1 A J 0.174489 0.891775 5 df1 B J 0.332001 0.568868 Then loop over each group of variable and plot the cumulative sum: c = ["blue", "purple", "red", "green", "pink"] for i, g in enumerate(dfall.groupby("variable")): ax = sns.barplot(data=g[1], x="index", y="vcs", hue="Name", color=c[i], zorder=-i, # so first bars stay on top edgecolor="k") ax.legend_.remove() # remove the redundant legends It lacks the legend that can be added easily I think. The problem is that instead of hatches (which can be added easily) to differentiate the dataframes we have a gradient of lightness, and it's a bit too light for the first one, and I don't really know how to change that without changing each rectangle one by one (as in the first solution). Tell me if you don't understand something in the code. Feel free to re-use this code which is under CC0.


Instruction:
How to read multiple lines of raw input?

I want to create a Python program which takes in multiple lines of user input. For example: This is a multilined input. It has multiple sentences. Each sentence is on a newline. How can I take in multiple lines of raw input?

Output: sentinel = '' # ends when this string is seen for line in iter(input, sentinel): pass # do things here To get every line as a string you can do: '\n'.join(iter(input, sentinel)) Python 2: '\n'.join(iter(raw_input, sentinel))


Instruction:
Difference between type(obj) and obj.__class__

What is the difference between type(obj) and obj.__class__? Is there ever a possibility of type(obj) is not obj.__class__? I want to write a function that works generically on the supplied objects, using a default value of 1 in the same type as another parameter. Which variation, #1 or #2 below, is going to do the right thing? def f(a, b=None): if b is None: b = type(a)(1) # #1 b = a.__class__(1) # #2

Output: type(obj) and type.__class__ do not behave the same for old style classes in Python 2: >>> class a(object): ... pass ... >>> class b(a): ... pass ... >>> class c: ... pass ... >>> ai = a() >>> bi = b() >>> ci = c() >>> type(ai) is ai.__class__ True >>> type(bi) is bi.__class__ True >>> type(ci) is ci.__class__ False >>> type(ci) <type 'instance'> >>> ci.__class__ <class __main__.c at 0x7f437cee8600> This is explained in the documentation: For an old-style class, the statement x.__class__ provides the class of x, but type(x) is always <type 'instance'>. This reflects the fact that all old-style instances, independent of their class, are implemented with a single built-in type, called instance.


Instruction:
How to count items in JSON data

How I can get the number of elements in node of JSON data? JSON: { "result":[ { "run":[ { "action":"stop" }, { "action":"start" }, { "action":"start" } ], "find":true } ] } I need to get the number of elements from node data['result'][0]['run']. It should be 3, but I can't find how to do it in Python.

Output: import json json_data = json.dumps({ "result":[ { "run":[ { "action":"stop" }, { "action":"start" }, { "action":"start" } ], "find": "true" } ] }) item_dict = json.loads(json_data) print len(item_dict['result'][0]['run']) Convert it into Python dictionary


Instruction:
What is the difference between C.UTF-8 and en_US.UTF-8 locales?

I'm migrating a Python application from an Ubuntu server with a en_US.UTF-8 locale to a new Debian server which comes with C.UTF-8 already set by default. I'm trying to understand if there could be any impact from this change.

Output: In general C is for computer, en_US is for people in the US who speak English (and other people who want the same behaviour). The for computer means that the strings are sometimes more standardized (but still in English), so an output of a program could be read from another program. With en_US, strings could be improved, alphabetic order could be improved (maybe by new rules of Chicago rules of style, etc.). So more user-friendly, but possibly less stable. Note: locales are not just for translation of strings, but also for collation: alphabetic order, numbers (e.g. thousand separator), currency (I think it is safe to predict that $ and 2 decimal digits will remain), months, day of weeks, etc. In your case, it is just the UTF-8 version of both locales. In general it should not matter. I usually prefer en_US.UTF-8, but usually it doesn't matter, and in your case (server app), it should only change log and error messages (if you use locale.setlocale(). You should handle client locales inside your app. Programs that read from other programs should set C before opening the pipe, so it should not really matter. As you see, probably it doesn't matter. You may also use the POSIX locale, also defined in Debian. You get the list of installed locales with locale -a. Note: Micro-optimization will prescribe C/C.UTF-8 locale: no translation of files (gettext), and simple rules on collation and number formatting, but this should be visible only on the server side.


Instruction:
Does Numpy automatically detect and use GPU?

I have a few basic questions about using Numpy with GPU (nvidia GTX 1080 Ti). I'm new to GPU, and would like to make sure I'm properly using the GPU to accelerate Numpy/Python. I searched on the internet for a while, but didn't find a simple tutorial that addressed my questions. I'd appreciate it if someone can give me some pointers: 1) Does Numpy/Python automatically detect the presence of GPU and utilize it to speed up matrix computation (e.g. numpy.multiply, numpy.linalg.inv, ... etc)? Or do I have code in a specific way to exploit the GPU for fast computation? 2) Can someone recommend a good tutorial/introductory material on using Numpy/Python with GPU (nvidia's)? Thanks a lot!

Output: Does Numpy/Python automatically detect the presence of GPU and utilize it to speed up matrix computation (e.g. numpy.multiply, numpy.linalg.inv, ... etc)? No. Or do I have code in a specific way to exploit the GPU for fast computation? Yes. Search for Numba, CuPy, Theano, PyTorch or PyCUDA for different paradigms for accelerating Python with GPUs.


Instruction:
Module imports and __init__.py

I am trying to understand what the best practices are with regards to Python's (v2.7) import mechanics. I have a project that has started to grow a bit and lets say my code is organized as follows: foo/ __init__.py Foo.py module1.py module2.py module3.py The package name is foo and underneath it I have module Foo.py which contains code for the class Foo. Hence I am using the same name for the package, module and class which might not be very clever to start with. __init__.py is empty and class Foo needs to import module1, module2 and module3 hence part of my Foo.py file looks like: # foo/Foo.py import module1 import module2 import module3 class Foo(object): def __init__(self): .... .... if __name__ == '__main__': foo_obj = Foo() However I later revisited this and I thought it would be better to have all imports in the __init__.py file. Hence my __init__.py now looks like: # foo/__init__.py import Foo import module1 import module2 import module3 .... .... and my Foo.py only needs to import foo: # foo/Foo.py import foo While this looks convenient since it is a one liner, I am a bit worried that it might be creating circular imports. What I mean is that when the script Foo.py is run it will import everything it can and then __init__.py will be called which will import Foo.py again (is that correct?). Additionally using the same name for package, module and class makes things more confusing. Does it make sense the way I have done it? Or am I asking for trouble?

Output: A couple things you could do to improve your organization, if only to adhere to some popular Python conventions and standards. If you search this topic, you will inevitably run across people recommending the PEP8 guidelines. These are the de facto canonical standards for organizing python code. Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged. Based on these guidelines, your project modules should be named like this: foo/ __init__.py foo.py module1.py module2.py module3.py I find it's generally best to avoid importing modules unnecessarily in __init__.py unless you're doing it for namespace reasons. For example, if you want the namespace for your package to look like this from foo import Foo instead of from foo.foo import Foo Then it makes sense to put from .foo import Foo in your __init__.py. As your package gets larger, some users may not want to use all of the sub-packages and modules, so it doesn't make sense to force the user to wait for all those modules to load by implicitly importing them in your __init__.py. Also, you have to consider whether you even want module1, module2, and module3 as part of your external API. Are they only used by Foo and not intended to be for end users? If they're only used internally, then don't include them in the __init__.py I'd also recommend using absolute or explicit relative imports for importing sub-modules. For example, in foo.py Absolute from foo import module1 from foo import module2 from foo import module3 Explicit Relative from . import module1 from . import module2 from . import module3 This will prevent any possible naming issues with other packages and modules. It will also make it easier if you decide to support Python3, since the implicit relative import syntax you're currently using is not supported in Python3. Also, files inside your package generally shouldn't contain a if __name__ == '__main__' This is because running a file as a script means it won't be considered part of the package that it belongs to, so it won't be able to make relative imports. The best way to provide executable scripts to users is by using the scripts or console_scripts feature of setuptools. The way you organize your scripts can be different depending on which method you use, but I generally organize mine like this: foo/ __init__.py foo.py ... scripts/ foo_script.py setup.py


Instruction:
How do I access my webcam in Python?

I would like to access my webcam from Python. I tried using the VideoCapture extension (tutorial), but that didn't work very well for me, I had to work around some problems such as it's a bit slow with resolutions >320x230, and sometimes it returns None for no apparent reason. Is there a better way to access my webcam from Python?

Output: OpenCV has support for getting data from a webcam, and it comes with Python wrappers by default, you also need to install numpy for the OpenCV Python extension (called cv2) to work. As of 2019, you can install both of these libraries with pip: pip install numpy pip install opencv-python More information on using OpenCV with Python. An example copied from Displaying webcam feed using opencv and python: import cv2 cv2.namedWindow("preview") vc = cv2.VideoCapture(0) if vc.isOpened(): # try to get the first frame rval, frame = vc.read() else: rval = False while rval: cv2.imshow("preview", frame) rval, frame = vc.read() key = cv2.waitKey(20) if key == 27: # exit on ESC break vc.release() cv2.destroyWindow("preview")


Instruction:
awscli not added to path after installation

I installed the aws cli according to the offical Amazon directions. sudo pip install awscli However, aws is nowhere to be found in my path. The installation seems to have been successful. There are a number of files located at /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/awscli, however there are no executables named aws. My python version is 3.3.4, my pip version is 1.5.4, and running this command on OS X 10.9. What could be wrong? Thanks!

Output: Improving the OP's Answer The OP answered their own question, but the exact location of the executable is more likely to be different than it is to be the same. So, let's break down WHY his solution worked so you can apply it to yourself. From the problem There are a number of files located at /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/awscli, however there are no executables named aws. From the solution The solution was to add /Library/Frameworks/Python.framework/Versions/3.3/bin to the my PATH. Let's learn something Compare those paths to find their commonality: /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/awscli /Library/Frameworks/Python.framework/Versions/3.3/bin Notice that they diverge at lib vs. bin. And consider that the OP stated, "there are no executables named aws." That brings us to our first learning lessons: Executables tend to not be in lib folders. Look for bin folders that share a common lineage. In this case I would have suggested looking for bin folders via: find /Library/Frameworks/Python.framework -type d -name bin But, if you are going to do that, you might as well just search for your executable via: find /Library/Frameworks/Python.framework -type f -perm -100 -name aws # the `-` in `perm -100` means not an exact match of 100 # but any octal that includes 100 But wait How did OP know to look in their /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/? The easiest answer is also our next learning lesson: Ask your python where things are installed. Here is how I do that: $ python -c 'import awscli; print(awscli)' <module 'awscli' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/awscli/__init__.pyc'> $ python3 -c 'import awscli; print(awscli)' <module 'awscli' from '/System/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/awscli/__init__.py'> I have 2 Pythons and neither of them use the same paths or even path patterns as the OP. Apply what we've learned $ find /System/Library/Frameworks/Python.framework -type d -name bin /System/Library/Frameworks/Python.framework/Versions/2.7/bin /System/Library/Frameworks/Python.framework/Versions/3.6/bin $ find /System/Library/Frameworks/Python.framework -type f -perm -100 -name aws /System/Library/Frameworks/Python.framework/Versions/2.7/bin/aws /System/Library/Frameworks/Python.framework/Versions/3.6/bin/aws As you can see, I have 2 bin folders and 2 aws executables. I probably want to use the Python3.6 version. However, if I'm doing local trial and error work for a remote system that uses the Python2.7 version, I'm going to want to use that. And this is exactly why I have 2 version installed.


Instruction:
Unittest tests order

How can I be sure of the unittest methods' order? Is the alphabetical or numeric prefixes the proper way? class TestFoo(TestCase): def test_1(self): ... def test_2(self): ... or class TestFoo(TestCase): def test_a(self): ... def test_b(self): ...

Output: You can disable it by setting sortTestMethodsUsing to None: import unittest unittest.TestLoader.sortTestMethodsUsing = None For pure unit tests, you folks are right; but for component tests and integration tests... I do not agree that you shall assume nothing about the state. What if you are testing the state? For example, your test validates that a service is auto-started upon installation. If in your setup, you start the service, then do the assertion, and then you are no longer testing the state, but you are testing the "service start" functionality. Another example is when your setup takes a long time or requires a lot of space and it just becomes impractical to run the setup frequently. Many developers tend to use "unit test" frameworks for component testing...so stop and ask yourself, am I doing unit testing or component testing?


Instruction:
DLL load failed error when importing cv2

I have installed opencv on my windows machine using python 3.6 without any issues, using: pip install opencv-python but when I try to import cv2 I get the following error ImportError: DLL load failed: The specified module could not be found. I have seen this post It says cv2 doesn't work with python 3 I was wondering if this has been fixed or if there is a way around it

Output: You can download the latest OpenCV 3.2.0 for Python 3.6 on Windows 32-bit or 64-bit machine, look for file starts withopencv_python‑3.2.0‑cp36‑cp36m, from this unofficial site. Then type below command to install it: pip install opencv_python‑3.2.0‑cp36‑cp36m‑win32.whl (32-bit version) pip install opencv_python‑3.2.0‑cp36‑cp36m‑win_amd64.whl (64-bit version) I think it would be easier. Update on 2017-09-15: OpenCV 3.3.0 wheel files are now available in the unofficial site and replaced OpenCV 3.2.0. Update on 2018-02-15: OpenCV 3.4.0 wheel files are now available in the unofficial site and replaced OpenCV 3.3.0. Update on 2018-06-19: OpenCV 3.4.1 wheel files are now available in the unofficial site with CPython 3.5/3.6/3.7 support, and replaced OpenCV 3.4.0. Update on 2018-10-03: OpenCV 3.4.3 wheel files are now available in the unofficial site with CPython 3.5/3.6/3.7 support, and replaced OpenCV 3.4.1. Update on 2019-01-30: OpenCV 4.0.1 wheel files are now available in the unofficial site with CPython 3.5/3.6/3.7 support. Update on 2019-06-10: OpenCV 3.4.6 and OpenCV 4.1.0 wheel files are now available in the unofficial site with CPython 3.5/3.6/3.7 support. Update on 2023-02-11: OpenCV 4.5.5 wheel files are now available in the unofficial site with CPython 3.7/3.8/3.9/3.10/3.11 support.


Instruction:
CSRF validation does not work on Django using HTTPS

I am developing an application which the frontend is an AngularJS API that makes requests to the backend API developed in Django Rest Framework. The frontend is on the domain: https://front.bluemix.net And my backend is on the domain: https://back.bluemix.net I am having problems making requests from the frontend API to the backend API. The error is this: Error: CSRF Failed: Referer checking failed - https://front.bluemix.net does not match any trusted origins. I am using CORS and I have already included the following lines in my settings.py in the Django backend API: ALLOWED_HOSTS = [] CORS_ALLOW_CREDENTIALS = True CORS_ORIGIN_ALLOW_ALL = True CORS_ALLOW_CREDENTIALS = True CSRF_TRUSTED_ORIGINS = ['https://front.bluemix.net/'] CORS_REPLACE_HTTPS_REFERER = True CSRF_COOKIE_DOMAIN = 'bluemix.net' CORS_ORIGIN_WHITELIST = ( 'https://front.bluemix.net/', 'front.bluemix.net', 'bluemix.net', ) Anyone knows how to solve this problem?

Output: Django 4.0 and above For Django 4.0 and above, CSRF_TRUSTED_ORIGINS must include scheme and host, e.g.: CSRF_TRUSTED_ORIGINS = ['https://front.bluemix.net'] Django 3.2 and lower For Django 3.2 and lower, CSRF_TRUSTED_ORIGINS must contain only the hostname, without a scheme: CSRF_TRUSTED_ORIGINS = ['front.bluemix.net'] You probably also need to put something in ALLOWED_HOSTS...


Instruction:
How to use rolling functions for GroupBy objects

I have a time series object grouped of the type <pandas.core.groupby.SeriesGroupBy object at 0x03F1A9F0>. grouped.sum() gives the desired result but I cannot get rolling_sum to work with the groupby object. Is there any way to apply rolling functions to groupby objects? For example: x = range(0, 6) id = ['a', 'a', 'a', 'b', 'b', 'b'] df = DataFrame(zip(id, x), columns = ['id', 'x']) df.groupby('id').sum() id x a 3 b 12 However, I would like to have something like: id x 0 a 0 1 a 1 2 a 3 3 b 3 4 b 7 5 b 12

Output: cumulative sum To answer the question directly, the cumsum method would produced the desired series: In [17]: df Out[17]: id x 0 a 0 1 a 1 2 a 2 3 b 3 4 b 4 5 b 5 In [18]: df.groupby('id').x.cumsum() Out[18]: 0 0 1 1 2 3 3 3 4 7 5 12 Name: x, dtype: int64 pandas rolling functions per group More generally, any rolling function can be applied to each group as follows (using the new .rolling method as commented by @kekert). Note that the return type is a multi-indexed series, which is different from previous (deprecated) pd.rolling_* methods. In [10]: df.groupby('id')['x'].rolling(2, min_periods=1).sum() Out[10]: id a 0 0.00 1 1.00 2 3.00 b 3 3.00 4 7.00 5 9.00 Name: x, dtype: float64 To apply the per-group rolling function and receive result in original dataframe order, transform should be used instead: In [16]: df.groupby('id')['x'].transform(lambda s: s.rolling(2, min_periods=1).sum()) Out[16]: 0 0 1 1 2 3 3 3 4 7 5 9 Name: x, dtype: int64 deprecated approach For reference, here's how the now deprecated pandas.rolling_mean behaved: In [16]: df.groupby('id')['x'].apply(pd.rolling_mean, 2, min_periods=1) Out[16]: 0 0.0 1 0.5 2 1.5 3 3.0 4 3.5 5 4.5


Instruction:
Create a new RGB OpenCV image using Python?

Using OpenCV in Python, how can I create a new RGB image? I don't want to load the image from a file, just create an empty image ready to work with.

Output: The new cv2 interface for Python integrates numpy arrays into the OpenCV framework, which makes operations much simpler as they are represented with simple multidimensional arrays. For example, your question would be answered with: import cv2 # Not actually necessary if you just want to create an image. import numpy as np blank_image = np.zeros((height,width,3), np.uint8) This initialises an RGB-image that is just black. Now, for example, if you wanted to set the left half of the image to blue and the right half to green , you could do so easily: blank_image[:,0:width//2] = (255,0,0) # (B, G, R) blank_image[:,width//2:width] = (0,255,0) If you want to save yourself a lot of trouble in future, as well as having to ask questions such as this one, I would strongly recommend using the cv2 interface rather than the older cv one. I made the change recently and have never looked back. You can read more about cv2 at the OpenCV Change Logs.


Instruction:
How to zip two 1d numpy array to 2d numpy array

I have two numpy 1d arrays, e.g: a = np.array([1,2,3,4,5]) b = np.array([6,7,8,9,10]) Then how can I get one 2d array [[1,6], [2,7], [3,8], [4,9], [5, 10]]?

Output: The answer lies in your question: np.array(list(zip(a,b))) Edit: Although my post gives the answer as requested by the OP, the conversion to list and back to NumPy array takes some overhead (noticeable for large arrays). Hence, dstack would be a computationally efficient alternative (ref. @zipa's answer). I was unaware of dstack at the time of posting this answer so credits to @zipa for introducing it to this post. Edit 2: As can be seen in the duplicate question, np.c_ is even shorter than np.dstack. >>> import numpy as np >>> a = np.arange(1, 6) >>> b = np.arange(6, 11) >>> >>> a array([1, 2, 3, 4, 5]) >>> b array([ 6, 7, 8, 9, 10]) >>> np.c_[a, b] array([[ 1, 6], [ 2, 7], [ 3, 8], [ 4, 9], [ 5, 10]])


Instruction:
Calling static method in python

I have a class Person and a static method in that class called call_person: class Person: def call_person(): print "hello person" In the python console I import the class Person and call Person.call_person(). But it is giving me error that says 'module' object has no attribute 'call_person'. Can anyone please let me know why I am getting this error?

Output: You need to do something like: class Person: @staticmethod def call_person(): print("hello person") # Calling static methods works on classes as well as instances of that class Person.call_person() # calling on class p = Person() p.call_person() # calling on instance of class Depending on what you want to do, a classmethod might be more appropriate: class Person: @classmethod def call_person(cls): print("hello person", cls) p = Person().call_person() # using classmethod on instance Person.call_person() # using classmethod on class The difference here is that in the second example, the class itself is passed as the first argument to the method (as opposed to a regular method where the instance is the first argument, or a staticmethod which doesn't receive any additional arguments). Now to answer your actual question. I'm betting that you aren't finding your method because you have put the class Person into a module Person.py. Then: import Person # Person class is available as Person.Person Person.Person.call_person() # this should work Person.Person().call_person() # this should work as well Alternatively, you might want to import the class Person from the module Person: from Person import Person Person.call_person() This all gets a little confusing as to what is a module and what is a class. Typically, I try to avoid giving classes the same name as the module that they live in. However, this is apparently not looked down on too much as the datetime module in the standard library contains a datetime class. Finally, it is worth pointing out that you don't need a class for this simple example: # Person.py def call_person(): print("Hello person") Now in another file, import it: import Person Person.call_person() # 'Hello person'


Instruction:
Check if a string in a Pandas DataFrame column is in a list of strings

If I have a frame like this frame = pd.DataFrame({ "a": ["the cat is blue", "the sky is green", "the dog is black"] }) and I want to check if any of those rows contain a certain word I just have to do this. frame["b"] = ( frame.a.str.contains("dog") | frame.a.str.contains("cat") | frame.a.str.contains("fish") ) frame["b"] outputs: 0 True 1 False 2 True Name: b, dtype: bool If I decide to make a list: mylist = ["dog", "cat", "fish"] How would I check that the rows contain a certain word in the list?

Output: frame = pd.DataFrame({'a' : ['the cat is blue', 'the sky is green', 'the dog is black']}) frame a 0 the cat is blue 1 the sky is green 2 the dog is black The str.contains method accepts a regular expression pattern: mylist = ['dog', 'cat', 'fish'] pattern = '|'.join(mylist) pattern 'dog|cat|fish' frame.a.str.contains(pattern) 0 True 1 False 2 True Name: a, dtype: bool Because regex patterns are supported, you can also embed flags: frame = pd.DataFrame({'a' : ['Cat Mr. Nibbles is blue', 'the sky is green', 'the dog is black']}) frame a 0 Cat Mr. Nibbles is blue 1 the sky is green 2 the dog is black pattern = '|'.join([f'(?i){animal}' for animal in mylist]) # python 3.6+ pattern '(?i)dog|(?i)cat|(?i)fish' frame.a.str.contains(pattern) 0 True # Because of the (?i) flag, 'Cat' is also matched to 'cat' 1 False 2 True


Instruction:
Delete a subplot

I'm trying to figure out a way of deleting (dynamically) subplots in matplotlib. I see they have a remove method, but I get the error NotImplementedError: cannot remove artist I'm surprised that I can't find this anywhere. Does anyone know how to do this? from matplotlib import pyplot as plt fig, axs = plt.subplots(1,3) axs[0].plot([1,2],[3,4]) axs[2].plot([0,1],[2,3]) plt.draw() plt.tight_layout()

Output: Use fig.delaxes or plt.delaxes to remove unwanted subplots fig, axs = plt.subplots(1,3) axs[0].plot([1,2],[3,4]) axs[2].plot([0,1],[2,3]) fig.delaxes(axs[1]) plt.draw() plt.tight_layout()


Instruction:
How are tuples unpacked in for loops?

I stumbled across the following code: for i, a in enumerate(attributes): labels.append(Label(root, text = a, justify = LEFT).grid(sticky = W)) e = Entry(root) e.grid(column=1, row=i) entries.append(e) entries[i].insert(INSERT,"text to insert") I don't understand the i, a bit, and searching for information on for didn't yield any useful results. When I try and experiment with the code I get the error: ValueError: need more than 1 value to unpack Does anyone know what it does, or a more specific term associated with it that I can google to learn more?

Output: You could google "tuple unpacking". This can be used in various places in Python. The simplest is in assignment: >>> x = (1,2) >>> a, b = x >>> a 1 >>> b 2 In a for-loop it works similarly. If each element of the iterable is a tuple, then you can specify two variables, and each element in the loop will be unpacked to the two. >>> x = [(1,2), (3,4), (5,6)] >>> for item in x: ... print "A tuple", item A tuple (1, 2) A tuple (3, 4) A tuple (5, 6) >>> for a, b in x: ... print "First", a, "then", b First 1 then 2 First 3 then 4 First 5 then 6 The enumerate function creates an iterable of tuples, so it can be used this way.


Instruction:
Strip whitespace from strings in a column

I am using python csvkit to compare 2 files like this: df1 = pd.read_csv('input1.csv', sep=',\s+', delimiter=',', encoding="utf-8") df2 = pd.read_csv('input2.csv', sep=',\s,', delimiter=',', encoding="utf-8") df3 = pd.merge(df1,df2, on='employee_id', how='right') df3.to_csv('output.csv', encoding='utf-8', index=False) Currently I am running the file through a script before hand that strips spaces from the employee_id column. An example of employee_ids: 37 78973 3 23787 2 22 3 123 Is there a way to get csvkit to do it and save me a step?

Output: You can strip() an entire Series in Pandas using .str.strip(): df1['employee_id'] = df1['employee_id'].str.strip() df2['employee_id'] = df2['employee_id'].str.strip() This will remove leading/trailing whitespaces on the employee_id column in both df1 and df2 Alternatively, modify the read_csv lines to use skipinitialspace=True df1 = pd.read_csv('input1.csv', sep=',\s+', delimiter=',', encoding="utf-8", skipinitialspace=True) df2 = pd.read_csv('input2.csv', sep=',\s,', delimiter=',', encoding="utf-8", skipinitialspace=True) It looks like you are attempting to remove spaces in a string containing numbers, which can be accomplished with pandas.Series.str.replace: df1['employee_id'] = df1['employee_id'].str.replace(" ", "") df2['employee_id'] = df2['employee_id'].str.replace(" ", "")


Instruction:
File not found error when launching a subprocess containing piped commands

I need to run the command date | grep -o -w '"+tz+"'' | wc -w using Python on my localhost. I am using subprocess module for the same and using the check_output method as I need to capture the output for the same. However it is throwing me an error : Traceback (most recent call last): File "test.py", line 47, in <module> check_timezone() File "test.py", line 40, in check_timezone count = subprocess.check_output(command) File "/usr/lib/python2.7/subprocess.py", line 537, in check_output process = Popen(stdout=PIPE, *popenargs, **kwargs) File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child raise child_exception- OSError: [Errno 2] No such file or directory

Output: You have to add shell=True to execute a shell command. check_output is trying to find an executable called: date | grep -o -w '"+tz+"'' | wc -w and cannot find it. (no idea why you removed the essential information from the error message). See the difference between: >>> subprocess.check_output('date | grep 1') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.4/subprocess.py", line 603, in check_output with Popen(*popenargs, stdout=PIPE, **kwargs) as process: File "/usr/lib/python3.4/subprocess.py", line 848, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.4/subprocess.py", line 1446, in _execute_child raise child_exception_type(errno_num, err_msg) FileNotFoundError: [Errno 2] No such file or directory: 'date | grep 1' And: >>> subprocess.check_output('date | grep 1', shell=True) b'gio 19 giu 2014, 14.15.35, CEST\n' Read the documentation about the Frequently Used Arguments for more information about the shell argument and how it changes the interpretation of the other arguments. Note that you should try to avoid using shell=True since spawning a shell can be a security hazard (even if you do not execute untrusted input attacks like Shellshock can still be performed!). The documentation for the subprocess module has a little section about replacing the shell pipeline. You can do so by spawning the two processes in python and use subprocess.PIPE: date_proc = subprocess.Popen(['date'], stdout=subprocess.PIPE) grep_proc = subprocess.check_output(['grep', '1'], stdin=date_proc.stdout, stdout=subprocess.PIPE) date_proc.stdout.close() output = grep_proc.communicate()[0] You can write some simple wrapper function to easily define pipelines: import subprocess from shlex import split from collections import namedtuple from functools import reduce proc_output = namedtuple('proc_output', 'stdout stderr') def pipeline(starter_command, *commands): if not commands: try: starter_command, *commands = starter_command.split('|') except AttributeError: pass starter_command = _parse(starter_command) starter = subprocess.Popen(starter_command, stdout=subprocess.PIPE) last_proc = reduce(_create_pipe, map(_parse, commands), starter) return proc_output(*last_proc.communicate()) def _create_pipe(previous, command): proc = subprocess.Popen(command, stdin=previous.stdout, stdout=subprocess.PIPE) previous.stdout.close() return proc def _parse(cmd): try: return split(cmd) except Exception: return cmd With this in place you can write pipeline('date | grep 1') or pipeline('date', 'grep 1') or pipeline(['date'], ['grep', '1'])


Instruction:
How to delete an Elasticsearch Index using Python?

I have Python 2.7 and Elasticsearch 2.1.1. I used the following to delete an index: es.delete(index='researchtest', doc_type='test') but this gives me this message: return func(*args, params=params, **kwargs) TypeError: delete() takes at least 4 arguments (4 given) I also tried this technique: es.delete_by_query( index='researchtest', doc_type='test', body='{"query":{"match_all":{}}}' ) but I get this message: AttributeError: 'Elasticsearch' object has no attribute 'delete_by_query' Has the API changed for 2.1.1 for python? https://elasticsearch-py.readthedocs.org/en/master/api.html#elasticsearch.client.IndicesClient.delete

Output: For ES 8+ use: from elasticsearch import Elasticsearch es = Elasticsearch() es.options(ignore_status=[400,404]).indices.delete(index='test-index') For older versions use this notation: from elasticsearch import Elasticsearch es = Elasticsearch() es.indices.delete(index='test-index', ignore=[400, 404])


Instruction:
Sieve of Eratosthenes - Finding Primes Python

Just to clarify, this is not a homework problem :) I wanted to find primes for a math application I am building & came across Sieve of Eratosthenes approach. I have written an implementation of it in Python. But it's terribly slow. For say, if I want to find all primes less than 2 million. It takes > 20 mins. (I stopped it at this point). How can I speed this up? def primes_sieve(limit): limitn = limit+1 primes = range(2, limitn) for i in primes: factors = range(i, limitn, i) for f in factors[1:]: if f in primes: primes.remove(f) return primes print primes_sieve(2000) UPDATE: I ended up doing profiling on this code & found that quite a lot of time was spent on removing an element from the list. Quite understandable considering it has to traverse the entire list (worst-case) to find the element & then remove it and then readjust the list (maybe some copy goes on?). Anyway, I chucked out list for dictionary. My new implementation - def primes_sieve1(limit): limitn = limit+1 primes = dict() for i in range(2, limitn): primes[i] = True for i in primes: factors = range(i,limitn, i) for f in factors[1:]: primes[f] = False return [i for i in primes if primes[i]==True] print primes_sieve1(2000000)

Output: You're not quite implementing the correct algorithm: In your first example, primes_sieve doesn't maintain a list of primality flags to strike/unset (as in the algorithm), but instead resizes a list of integers continuously, which is very expensive: removing an item from a list requires shifting all subsequent items down by one. In the second example, primes_sieve1 maintains a dictionary of primality flags, which is a step in the right direction, but it iterates over the dictionary in undefined order, and redundantly strikes out factors of factors (instead of only factors of primes, as in the algorithm). You could fix this by sorting the keys, and skipping non-primes (which already makes it an order of magnitude faster), but it's still much more efficient to just use a list directly. The correct algorithm (with a list instead of a dictionary) looks something like: def primes_sieve2(limit): a = [True] * limit # Initialize the primality list a[0] = a[1] = False for (i, isprime) in enumerate(a): if isprime: yield i for n in range(i*i, limit, i): # Mark factors non-prime a[n] = False (Note that this also includes the algorithmic optimization of starting the non-prime marking at the prime's square (i*i) instead of its double.)


Instruction:
How can I use f-string with a variable, not with a string literal?

I want to use f-string with my string variable, not with string defined with a string literal, "...". Here is my code: name=["deep","mahesh","nirbhay"] user_input = r"certi_{element}" # this string I ask from user for element in name: print(f"{user_input}") This code gives output: certi_{element} certi_{element} certi_{element} But I want: certi_{deep} certi_{mahesh} certi_{nirbhay} How can I do this? See also: How to postpone/defer the evaluation of f-strings?

Output: If you define: def fstr(template): return eval(f"f'{template}'") Then you can do: name=["deep","mahesh","nirbhay"] user_input = r"certi_{element}" # this string i ask from user for element in name: print(fstr(user_input)) Which gives as output: certi_deep certi_mahesh certi_nirbhay But be aware that users can use expressions in the template, like e.g.: import os # assume you have used os somewhere user_input = r"certi_{os.environ}" for element in name: print(fstr(user_input)) You definitely don't want this! Therefore, a much safer option is to define: def fstr(template, **kwargs): return eval(f"f'{template}'", kwargs) Arbitrary code is no longer possible, but users can still use string expressions like: user_input = r"certi_{element.upper()*2}" for element in name: print(fstr(user_input, element=element)) Gives as output: certi_DEEPDEEP certi_MAHESHMAHESH certi_NIRBHAYNIRBHAY Which may be desired in some cases.


Instruction:
How to get the list of options that Python was compiled with?

You can compile Python in various ways. I'd like to find out with which options my Python was compiled. Concrete use-case: was my Python compiled with readline? I know I can see this by doing "import readline", but I'd like to see a list of compilation setting for my Python binary. Edit: I mean the Python executable and not source code written by myself.

Output: There is a module to see the system config import sysconfig print(sysconfig.get_config_vars()) It offers an interface to get individual variables as well. sysconfig.get_config_var('HAVE_LIBREADLINE') It is also accessible via shell: python3 -m sysconfig | less Edit: before python2.7, you have to use import distutils.sysconfig print distutils.sysconfig.get_config_vars()


Instruction:
Passing command line arguments to argv in jupyter/ipython notebook

I'm wondering if it's possible to populate sys.argv (or some other structure) with command line arguments in a jupyter/ipython notebook, similar to how it's done through a python script. For instance, if I were to run a python script as follows: python test.py False Then sys.argv would contain the argument False. But if I run a jupyter notebook in a similar manner: jupyter notebook test.ipynb False Then the command line argument gets lost. Is there any way to access this argument from within the notebook itself?

Output: If the goal is to run a notebook with configurable arguments passed from commandline, I think the easiest way is to use environment variables, like this: NB_ARGS=some_args jupyter nbconvert --execute --to html --template full some_notebook.ipynb Then in the notebook, you can import os and use os.environ['NB_ARGS']. The variable value can be some text that contains key-value pairs or json for example.


Instruction:
Gunicorn worker terminated with signal 9

I am running a Flask application and hosting it on Kubernetes from a Docker container. Gunicorn is managing workers that reply to API requests. The following warning message is a regular occurrence, and it seems like requests are being canceled for some reason. On Kubernetes, the pod is showing no odd behavior or restarts and stays within 80% of its memory and CPU limits. [2021-03-31 16:30:31 +0200] [1] [WARNING] Worker with pid 26 was terminated due to signal 9 How can we find out why these workers are killed?

Output: I encountered the same warning message. [WARNING] Worker with pid 71 was terminated due to signal 9 I came across this faq, which says that "A common cause of SIGKILL is when OOM killer terminates a process due to low memory condition." I used dmesg realized that indeed it was killed because it was running out of memory. Out of memory: Killed process 776660 (gunicorn)


Instruction:
Python: Typehints for argparse.Namespace objects

Is there a way to have Python static analyzers (e.g. in PyCharm, other IDEs) pick up on Typehints on argparse.Namespace objects? Example: parser = argparse.ArgumentParser() parser.add_argument('--somearg') parsed = parser.parse_args(['--somearg','someval']) # type: argparse.Namespace the_arg = parsed.somearg # <- Pycharm complains that parsed object has no attribute 'somearg' If I remove the type declaration in the inline comment, PyCharm doesn't complain, but it also doesn't pick up on invalid attributes. For example: parser = argparse.ArgumentParser() parser.add_argument('--somearg') parsed = parser.parse_args(['--somearg','someval']) # no typehint the_arg = parsed.somaerg # <- typo in attribute, but no complaint in PyCharm. Raises AttributeError when executed. Any ideas? Update Inspired by Austin's answer below, the simplest solution I could find is one using namedtuples: from collections import namedtuple ArgNamespace = namedtuple('ArgNamespace', ['some_arg', 'another_arg']) parser = argparse.ArgumentParser() parser.add_argument('--some-arg') parser.add_argument('--another-arg') parsed = parser.parse_args(['--some-arg', 'val1', '--another-arg', 'val2']) # type: ArgNamespace x = parsed.some_arg # good... y = parsed.another_arg # still good... z = parsed.aint_no_arg # Flagged by PyCharm! While this is satisfactory, I still don't like having to repeat the argument names. If the argument list grows considerably, it will be tedious updating both locations. What would be ideal is somehow extracting the arguments from the parser object like the following: parser = argparse.ArgumentParser() parser.add_argument('--some-arg') parser.add_argument('--another-arg') MagicNamespace = parser.magically_extract_namespace() parsed = parser.parse_args(['--some-arg', 'val1', '--another-arg', 'val2']) # type: MagicNamespace I haven't been able to find anything in the argparse module that could make this possible, and I'm still unsure if any static analysis tool could be clever enough to get those values and not bring the IDE to a grinding halt. Still searching... Update 2 Per hpaulj's comment, the closest thing I could find to the method described above that would "magically" extract the attributes of the parsed object is something that would extract the dest attribute from each of the parser's _actions.: parser = argparse.ArgumentParser() parser.add_argument('--some-arg') parser.add_argument('--another-arg') MagicNamespace = namedtuple('MagicNamespace', [act.dest for act in parser._actions]) parsed = parser.parse_args(['--some-arg', 'val1', '--another-arg', 'val2']) # type: MagicNamespace But this still does not cause attribute errors to get flagged in static analysis. This is true also true if I pass namespace=MagicNamespace in the parser.parse_args call.

Output: Typed argument parser was made for exactly this purpose. It wraps argparse. Your example is implemented as: from tap import Tap class ArgumentParser(Tap): somearg: str parsed = ArgumentParser().parse_args(['--somearg', 'someval']) the_arg = parsed.somearg Here's a picture of it in action. It's on PyPI and can be installed with: pip install typed-argument-parser Full disclosure: I'm one of the creators of this library.


Instruction:
pandas concat generates nan values

I am curious why a simple concatenation of two dataframes in pandas: initId.shape # (66441, 1) initId.isnull().sum() # 0 ypred.shape # (66441, 1) ypred.isnull().sum() # 0 of the same shape and both without NaN values foo = pd.concat([initId, ypred], join='outer', axis=1) foo.shape # (83384, 2) foo.isnull().sum() # 16943 can result in a lot of NaN values if joined. How can I fix this problem and prevent NaN values being introduced? Trying to reproduce it like aaa = pd.DataFrame([0,1,0,1,0,0], columns=['prediction']) bbb = pd.DataFrame([0,0,1,0,1,1], columns=['groundTruth']) pd.concat([aaa, bbb], axis=1) failed e.g. worked just fine as no NaN values were introduced.

Output: I think there is problem with different index values, so where concat cannot align get NaN: aaa = pd.DataFrame([0,1,0,1,0,0], columns=['prediction'], index=[4,5,8,7,10,12]) print(aaa) prediction 4 0 5 1 8 0 7 1 10 0 12 0 bbb = pd.DataFrame([0,0,1,0,1,1], columns=['groundTruth']) print(bbb) groundTruth 0 0 1 0 2 1 3 0 4 1 5 1 print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 0 NaN 0.0 1 NaN 0.0 2 NaN 1.0 3 NaN 0.0 4 0.0 1.0 5 1.0 1.0 7 1.0 NaN 8 0.0 NaN 10 0.0 NaN 12 0.0 NaN Solution is reset_index if indexes values are not necessary: aaa.reset_index(drop=True, inplace=True) bbb.reset_index(drop=True, inplace=True) print(aaa) prediction 0 0 1 1 2 0 3 1 4 0 5 0 print(bbb) groundTruth 0 0 1 0 2 1 3 0 4 1 5 1 print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 0 0 0 1 1 0 2 0 1 3 1 0 4 0 1 5 0 1 EDIT: If need same index like aaa and length of DataFrames is same use: bbb.index = aaa.index print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 4 0 0 5 1 0 8 0 1 7 1 0 10 0 1 12 0 1


Instruction:
Why is there no first(iterable) built-in function in Python?

I'm wondering if there's a reason that there's no first(iterable) in the Python built-in functions, somewhat similar to any(iterable) and all(iterable) (it may be tucked in a stdlib module somewhere, but I don't see it in itertools). first would perform a short-circuit generator evaluation so that unnecessary (and a potentially infinite number of) operations can be avoided; i.e. def identity(item): return item def first(iterable, predicate=identity): for item in iterable: if predicate(item): return item raise ValueError('No satisfactory value found') This way you can express things like: denominators = (2, 3, 4, 5) lcd = first(i for i in itertools.count(1) if all(i % denominators == 0 for denominator in denominators)) Clearly you can't do list(generator)[0] in that case, since the generator doesn't terminate. Or if you have a bunch of regexes to match against (useful when they all have the same groupdict interface): match = first(regex.match(big_text) for regex in regexes) You save a lot of unnecessary processing by avoiding list(generator)[0] and short-circuiting on a positive match.

Output: In Python 2, if you have an iterator, you can just call its next method. Something like: >>> (5*x for x in xrange(2,4)).next() 10 In Python 3, you can use the next built-in with an iterator: >>> next(5*x for x in range(2,4)) 10


Instruction:
SSL backend error when using OpenSSL

I was trying to install pycurl in a virtualenv using pip and I got this error ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other) I read some documentation saying that "To fix this, you need to tell setup.py what SSL backend is used" (source) although I am not sure how to do this since I installed pycurl using pip. How can I specify the SSL backend when installing pycurl with pip? Thanks

Output: for most people After reading their INSTALLATION file, I was able to solve my problem by setting an environment variable and did a reinstall # remove existing `pycurl` installation pip uninstall pycurl # export variable with your link-time ssl backend (which is openssl above) export PYCURL_SSL_LIBRARY=openssl # then, re-install `pycurl` with **no cache** pip install pycurl --no-cache-dir There could be other solution out there but this works perfectly for me on a virtualenv and pip installation. Some people have a different error message complaining about nss instead of openssl ImportError: pycurl: libcurl link-time ssl backend (nss) (the key part is nss) so you have to do something different for this error message: pip uninstall pycurl pip install --no-cache-dir --compile --compile-options="--with-nss" pycurl


Instruction:
Resize an image without distortion OpenCV

I am using Python 3 and latest version of OpenCV. I am trying to resize an image using the resize function provided, but after resizing, the image is very distorted. Code: import cv2 file = "/home/tanmay/Desktop/test_image.png" img = cv2.imread(file , 0) print(img.shape) cv2.imshow('img' , img) k = cv2.waitKey(0) if k == 27: cv2.destroyWindow('img') resize_img = cv2.resize(img , (28 , 28)) cv2.imshow('img' , resize_img) x = cv2.waitKey(0) if x == 27: cv2.destroyWindow('img') The original image is 480 x 640 (RGB, therefore I passed the 0 to get it to grayscale). Is there any way I can resize it and avoid the distortion using OpenCV or any other library perhaps? I intend to make a handwritten digit recogniser and I have trained my neural network using the MNIST data, therefore I need the image to be 28x28.

Output: You may try below. The function will keep the aspect rate of the original image. def image_resize(image, width = None, height = None, inter = cv2.INTER_AREA): # initialize the dimensions of the image to be resized and # grab the image size dim = None (h, w) = image.shape[:2] # if both the width and height are None, then return the # original image if width is None and height is None: return image # check to see if the width is None if width is None: # calculate the ratio of the height and construct the # dimensions r = height / float(h) dim = (int(w * r), height) # otherwise, the height is None else: # calculate the ratio of the width and construct the # dimensions r = width / float(w) dim = (width, int(h * r)) # resize the image resized = cv2.resize(image, dim, interpolation = inter) # return the resized image return resized Here is an example usage. image = image_resize(image, height = 800)


Instruction:
Generate pydantic model from a dict

Is there a straight-forward approach to generate a Pydantic model from a dictionary? Here is a sample of the data I have. { 'id': '424c015f-7170-4ac5-8f59-096b83fe5f5806082020', 'contacts': [{ 'displayName': 'Norma Fisher', 'id': '544aa395-0e63-4f9a-8cd4-767b3040146d' }], 'startTime': '2020-06-08T09:38:00+00:00' } Expecting a model similar to ... class NewModel(BaseModel): id: str contacts: list startTime: str

Output: In Pydantic 2, you can use MyModel.model_validate(my_dict) to generate a model from a dictionary. According to the documentation – this is very similar to the __init__ method of the model, except it takes a dict rather than keyword arguments. If you're Pydantic 1, the method is parse_obj instead.


Instruction:
Converting html to text with Python

I am trying to convert an html block to text using Python. Input:

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa

Consectetuer adipiscing elit. Some Link Aenean commodo ligula eget dolor. Aenean massa

Aenean massa.Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa

Consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa

 Desired output: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa Consectetuer adipiscing elit. Some Link Aenean commodo ligula eget dolor. Aenean massa Aenean massa.Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa Consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa I tried the html2text module without much success: #!/usr/bin/env python import urllib2 import html2text from BeautifulSoup import BeautifulSoup soup = BeautifulSoup(urllib2.urlopen('http://example.com/page.html').read()) txt = soup.find('div', {'class' : 'body'}) print(html2text.html2text(txt)) The txt object produces the html block above. I'd like to convert it to text and print it on the screen.

Output: soup.get_text() outputs what you want: from bs4 import BeautifulSoup soup = BeautifulSoup(html) print(soup.get_text()) output: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa Consectetuer adipiscing elit. Some Link Aenean commodo ligula eget dolor. Aenean massa Aenean massa.Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa Consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa To keep newlines: print(soup.get_text('\n')) To be identical to your example, you can replace a newline with two newlines: soup.get_text().replace('\n','\n\n')


Instruction:
Keras - Plot training, validation and test set accuracy

I want to plot the output of this simple neural network: model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) history = model.fit(x_test, y_test, nb_epoch=10, validation_split=0.2, shuffle=True) model.test_on_batch(x_test, y_test) model.metrics_names I have plotted accuracy and loss of training and validation: print(history.history.keys()) # "Accuracy" plt.plot(history.history['acc']) plt.plot(history.history['val_acc']) plt.title('model accuracy') plt.ylabel('accuracy') plt.xlabel('epoch') plt.legend(['train', 'validation'], loc='upper left') plt.show() # "Loss" plt.plot(history.history['loss']) plt.plot(history.history['val_loss']) plt.title('model loss') plt.ylabel('loss') plt.xlabel('epoch') plt.legend(['train', 'validation'], loc='upper left') plt.show() Now I want to add and plot test set's accuracy from model.test_on_batch(x_test, y_test), but from model.metrics_names I obtain the same value 'acc' utilized for plotting accuracy on training data plt.plot(history.history['acc']). How could I plot test set's accuracy?

Output: You could do it this way also .... regressor.compile(optimizer = 'adam', loss = 'mean_squared_error',metrics=['accuracy']) earlyStopCallBack = EarlyStopping(monitor='loss', patience=3) history=regressor.fit(X_train, y_train, validation_data=(X_test, y_test), epochs = EPOCHS, batch_size = BATCHSIZE, callbacks=[earlyStopCallBack]) For the plotting - I like plotly ... so import plotly.graph_objects as go from plotly.subplots import make_subplots # Create figure with secondary y-axis fig = make_subplots(specs=[[{"secondary_y": True}]]) # Add traces fig.add_trace( go.Scatter( y=history.history['val_loss'], name="val_loss"), secondary_y=False, ) fig.add_trace( go.Scatter( y=history.history['loss'], name="loss"), secondary_y=False, ) fig.add_trace( go.Scatter( y=history.history['val_accuracy'], name="val accuracy"), secondary_y=True, ) fig.add_trace( go.Scatter( y=history.history['accuracy'], name="val accuracy"), secondary_y=True, ) # Add figure title fig.update_layout( title_text="Loss/Accuracy of LSTM Model" ) # Set x-axis title fig.update_xaxes(title_text="Epoch") # Set y-axes titles fig.update_yaxes(title_text="primary Loss", secondary_y=False) fig.update_yaxes(title_text="secondary Accuracy", secondary_y=True) fig.show() Nothing wrong with either of the proceeding methods. Please note the Plotly graph has two scales , 1 for loss the other for accuracy.


Instruction:
Permutations with unique values

itertools.permutations generates where its elements are treated as unique based on their position, not on their value. So basically I want to avoid duplicates like this: >>> list(itertools.permutations([1, 1, 1])) [(1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1)] Filtering afterwards is not possible because the amount of permutations is too large in my case. Does anybody know of a suitable algorithm for this? What I basically want is the following: x = itertools.product((0, 1, 'x'), repeat=X) x = sorted(x, key=functools.partial(count_elements, elem='x')) which is not possible because sorted creates a list and the output of itertools.product is too large.

Output: class unique_element: def __init__(self,value,occurrences): self.value = value self.occurrences = occurrences def perm_unique(elements): eset=set(elements) listunique = [unique_element(i,elements.count(i)) for i in eset] u=len(elements) return perm_unique_helper(listunique,[0]*u,u-1) def perm_unique_helper(listunique,result_list,d): if d < 0: yield tuple(result_list) else: for i in listunique: if i.occurrences > 0: result_list[d]=i.value i.occurrences-=1 for g in perm_unique_helper(listunique,result_list,d-1): yield g i.occurrences+=1 a = list(perm_unique([1,1,2])) print(a) result: [(2, 1, 1), (1, 2, 1), (1, 1, 2)] EDIT (how this works): I rewrote the above program to be longer but more readable. I usually have a hard time explaining how something works, but let me try. In order to understand how this works, you have to understand a similar but simpler program that would yield all permutations with repetitions. def permutations_with_replacement(elements,n): return permutations_helper(elements,[0]*n,n-1)#this is generator def permutations_helper(elements,result_list,d): if d<0: yield tuple(result_list) else: for i in elements: result_list[d]=i all_permutations = permutations_helper(elements,result_list,d-1)#this is generator for g in all_permutations: yield g This program is obviously much simpler: d stands for depth in permutations_helper and has two functions. One function is the stopping condition of our recursive algorithm, and the other is for the result list that is passed around. Instead of returning each result, we yield it. If there were no function/operator yield we would have to push the result in some queue at the point of the stopping condition. But this way, once the stopping condition is met, the result is propagated through all stacks up to the caller. That is the purpose of for g in perm_unique_helper(listunique,result_list,d-1): yield g so each result is propagated up to caller. Back to the original program: we have a list of unique elements. Before we can use each element, we have to check how many of them are still available to push onto result_list. Working with this program is very similar to permutations_with_replacement. The difference is that each element cannot be repeated more times than it is in perm_unique_helper.


Instruction:
How do I resolve "No module named 'frontend'" when trying to use PyMuPDF/fitz?

I have installed PymuPDF/fitz because am trying to extract images from PDF files. However, upon running the code below, I am seeing No module named 'frontend'. import fitz doc = fitz.open(pdf_path) for i in range(len(doc)): for img in doc.getPageImageList(i): xref = img[0] pix = fitz.Pixmap(doc, xref) if pix.n < 5: # this is GRAY or RGB pix.writePNG("p%s-%s.png" % (i, xref)) else: # CMYK: convert to RGB first pix1 = fitz.Pixmap(fitz.csRGB, pix) pix1.writePNG("p%s-%s.png" % (i, xref)) pix1 = None pix = None I have searched but there isn't single report of this kind of error. I have installed PyMuPDF, muPDF and fitz modules. Here is the error in full: Traceback (most recent call last): File "/home/waqar/PycharmProjects/predator/ExtractFileImage.py", line 1, in <module> import fitz File "/home/waqar/anaconda3/envs/retinanet/lib/python3.6/site-packages/fitz/__init__.py", line 1, in <module> from frontend import * ModuleNotFoundError: No module named 'frontend'

Output: I've solved it by: pip install PyMuPDF This will actually allow the import of the fitz you appear to want. (There's another fitz, which is probably not what you want if you're manipulating PDF files.) NOTE: If you get RuntimeError: Directory 'static/' does not exist after install than do: pip uninstall fitz for more info see: raise RuntimeError(f"Directory '{directory}' does not exist") RuntimeError: Directory 'static/' does not exist from import fitz


Instruction:
Clone an image in cv2 Python

I'm new to OpenCV. What is the Python function which act the same as cv::clone() in C++? I just try to get a rect by rectImg = img[10:20, 10:20] but when I draw a line on it, I find the line appear both on img and the rectImage, so, how can I get this done?

Output: Abid Rahman K's answer is correct, but you say that you are using cv2 which inherently uses NumPy arrays. So, to make a complete different copy of say "myImage": newImage = myImage.copy() The above is enough. There isn't any need to import NumPy (numpy).


Instruction:
How do I display a single image in PyTorch?

How do I display a PyTorch Tensor of shape (3, 224, 224) representing a 224x224 RGB image? Using plt.imshow(image) gives the error: TypeError: Invalid dimensions for image data

Output: Given a Tensor representing the image, use .permute() to put the channels as the last dimension when passing them to matplotlib: plt.imshow(tensor_image.permute(1, 2, 0)) Note: permute does not copy or allocate memory, and from_numpy() doesn't either.


Instruction:
Positioning the colorbar

I have a matplotlib plot with a colorbar attached. I want to position the colorbar so that it is horizontal, and underneath my plot. I have almost done this via the following: plt.colorbar(orientation="horizontal",fraction=0.07,anchor=(1.0,0.0)) But the colorbar is still overlapping with the plot slightly (and the labels of the x axis). I want to move the colorbar further down, but I can't figure out how to do it.

Output: Edit: Updated for matplotlib version >= 3. Three great ways to do this have already been shared in this answer. The matplotlib documentation advises to use inset_locator. This would work as follows: import matplotlib.pyplot as plt from mpl_toolkits.axes_grid1.inset_locator import inset_axes import numpy as np rng = np.random.default_rng(1) fig, ax = plt.subplots(figsize=(4,4)) im = ax.imshow(rng.random((11, 16))) ax.set_xlabel("x label") axins = inset_axes(ax, width="100%", height="5%", loc='lower center', borderpad=-5 ) fig.colorbar(im, cax=axins, orientation="horizontal")


Instruction:
creating over 20 unique legend colors using matplotlib

I am plotting 20 different lines on a single plot using matplotlib. I use a for loop for plotting and label every line with its key and then use the legend function for key in dict.keys(): plot(x,dict[key], label = key) graph.legend() But using this way, the graph repeats a lot of colors in the legend. Is there any way to ensure a unique color is assigned to each line using matplotlib and over 20 lines? thanks

Output: The answer to your question is related to two other SO questions. The answer to How to pick a new color for each plotted line within a figure in matplotlib? explains how to define the default list of colors that is cycled through to pick the next color to plot. This is done with the Axes.set_color_cycle method. You want to get the correct list of colors though, and this is most easily done using a color map, as is explained in the answer to this question: Create a color generator from given colormap in matplotlib. There a color map takes a value from 0 to 1 and returns a color. So for your 20 lines, you want to cycle from 0 to 1 in steps of 1/20. Specifically you want to cycle form 0 to 19/20, because 1 maps back to 0. This is done in this example: import matplotlib.pyplot as plt import numpy as np NUM_COLORS = 20 cm = plt.get_cmap('gist_rainbow') fig = plt.figure() ax = fig.add_subplot(111) ax.set_prop_cycle(color=[cm(1.*i/NUM_COLORS) for i in range(NUM_COLORS)]) for i in range(NUM_COLORS): ax.plot(np.arange(10)*(i+1)) fig.savefig('moreColors.png') plt.show() This is the resulting figure: Alternative, better (debatable) solution There is an alternative way that uses a ScalarMappable object to convert a range of values to colors. The advantage of this method is that you can use a non-linear Normalization to convert from line index to actual color. The following code produces the same exact result: import matplotlib.pyplot as plt import matplotlib.cm as mplcm import matplotlib.colors as colors import numpy as np NUM_COLORS = 20 cm = plt.get_cmap('gist_rainbow') cNorm = colors.Normalize(vmin=0, vmax=NUM_COLORS-1) scalarMap = mplcm.ScalarMappable(norm=cNorm, cmap=cm) fig = plt.figure() ax = fig.add_subplot(111) # old way: #ax.set_prop_cycle(color=[cm(1.*i/NUM_COLORS) for i in range(NUM_COLORS)]) # new way: ax.set_prop_cycle(color=[scalarMap.to_rgba(i) for i in range(NUM_COLORS)]) for i in range(NUM_COLORS): ax.plot(np.arange(10)*(i+1)) fig.savefig('moreColors.png') plt.show()


Instruction:
pymongo.errors.CursorNotFound: cursor id '...' not valid at server

I am trying to fetch some ids that exist in a mongo database with the following code: client = MongoClient('xx.xx.xx.xx', xxx) db = client.test_database db = client['...'] collection = db.test_collection collection = db["..."] for cursor in collection.find({ "$and" : [{ "followers" : { "$gt" : 2000 } }, { "followers" : { "$lt" : 3000 } }, { "list_followers" : { "$exists" : False } }] }): print cursor['screenname'] print cursor['_id']['uid'] id = cursor['_id']['uid'] However, after a short while, I am receive this error: pymongo.errors.CursorNotFound: cursor id '...' not valid at server. I found this article which refers to that problem. Nevertheless it is not clear to me which solution to take. Is it possible to use find().batch_size(30)? What exactly does the above command do? Can I take all the database ids using batch_size?

Output: You're getting this error because the cursor is timing out on the server (after 10 minutes of inactivity). From the pymongo documentation: Cursors in MongoDB can timeout on the server if they’ve been open for a long time without any operations being performed on them. This can lead to an CursorNotFound exception being raised when attempting to iterate the cursor. When you call the collection.find method it queries a collection and it returns a cursor to the documents. To get the documents you iterate the cursor. When you iterate over the cursor the driver is actually making requests to the MongoDB server to fetch more data from the server. The amount of data returned in each request is set by the batch_size() method. From the documentation: Limits the number of documents returned in one batch. Each batch requires a round trip to the server. It can be adjusted to optimize performance and limit data transfer. Setting the batch_size to a lower value will help you with the timeout errors errors, but it will increase the number of times you're going to get access the MongoDB server to get all the documents. The default batch size: For most queries, the first batch returns 101 documents or just enough documents to exceed 1 megabyte. Batch size will not exceed the maximum BSON document size (16 MB). There is no universal "right" batch size. You should test with different values and see what is the appropriate value for your use case i.e. how many documents can you process in a 10 minute window. The last resort will be that you set no_cursor_timeout=True. But you need to be sure that the cursor is closed after you finish processing the data. How to avoid it without try/except: cursor = collection.find( {"x": 1}, no_cursor_timeout=True ) for doc in cursor: # do something with doc cursor.close()


Instruction:
What is a "method" in Python?

Can anyone, please, explain to me in very simple terms what a "method" is in Python? The thing is in many Python tutorials for beginners this word is used in such way as if the beginner already knew what a method is in the context of Python. While I am of course familiar with the general meaning of this word, I have no clue what this term means in Python. So, please, explain to me what the "Pythonian" method is all about. Some very simple example code would be very much appreciated as a picture is worth thousand words.

Output: It's a function which is a member of a class: class C: def my_method(self): print("I am a C") c = C() c.my_method() # Prints("I am a C") Simple as that! (There are also some alternative kinds of method, allowing you to control the relationship between the class and the function. But I'm guessing from your question that you're not asking about that, but rather just the basics.)


Instruction:
Type of compiled regex object in python

What is the type of the compiled regular expression in python? In particular, I want to evaluate isinstance(re.compile(''), ???) to be true, for introspection purposes. One solution I had was, have some global constant REGEX_TYPE = type(re.compile('')), but it doesn't seem very elegant. EDIT: The reason I want to do this is because I have list of strings and compiled regex objects. I want to "match" a string against list, by for each string in the list, try to check for string equality. for each regex in the list, try to check whether the string matches the given pattern. and the code that I came up with was: for allowed in alloweds: if isinstance(allowed, basestring) and allowed == input: ignored = False break elif isinstance(allowed, REGEX_TYPE) and allowed.match(input): ignored = False break

Output: When the type of something isn't well specified, there's nothing wrong with using the type builtin to discover the answer at runtime: >>> import re >>> retype = type(re.compile('hello, world')) >>> isinstance(re.compile('goodbye'), retype) True >>> isinstance(12, retype) False >>> Discovering the type at runtime protects you from having to access private attributes and against future changes to the return type. There's nothing inelegant about using type here, though there may be something inelegant about wanting to know the type at all. That said, with the passage of time, the context of this question has shifted. With contemporary versions of Python, the return type of re.compile is now re.Pattern. The general question about what to do if the type of something is not well-specified is still valid but in this particular case, the type of re.compile(...) is now well-specified.


Instruction:
How to assign a value to a TensorFlow variable?

I am trying to assign a new value to a tensorflow variable in python. import tensorflow as tf import numpy as np x = tf.Variable(0) init = tf.initialize_all_variables() sess = tf.InteractiveSession() sess.run(init) print(x.eval()) x.assign(1) print(x.eval()) But the output I get is 0 0 So the value has not changed. What am I missing?

Output: In TF1, the statement x.assign(1) does not actually assign the value 1 to x, but rather creates a tf.Operation that you have to explicitly run to update the variable.* A call to Operation.run() or Session.run() can be used to run the operation: assign_op = x.assign(1) sess.run(assign_op) # or `assign_op.op.run()` print(x.eval()) # ==> 1 (* In fact, it returns a tf.Tensor, corresponding to the updated value of the variable, to make it easier to chain assignments.) However, in TF2 x.assign(1) will now assign the value eagerly: x.assign(1) print(x.numpy()) # ==> 1


Instruction:
Pytest monkeypatch isn't working on imported function

Suppose there are two packages in a project: some_package and another_package. # some_package/foo.py: def bar(): print('hello') # another_package/function.py from some_package.foo import bar def call_bar(): # ... code ... bar() # ... code ... I want to test another_package.function.call_bar mocking out some_package.foo.bar because it has some network I/O I want to avoid. Here is a test: # tests/test_bar.py from another_package.function import call_bar def test_bar(monkeypatch): monkeypatch.setattr('some_package.foo.bar', lambda: print('patched')) call_bar() assert True To my surprise it outputs hello instead of patched. I tried to debug this thing, putting an IPDB breakpoint in the test. When I manually import some_package.foo.bar after the breakpoint and call bar() I get patched. On my real project the situation is even more interesting. If I invoke pytest in the project root my function isn't patched, but when I specify tests/test_bar.py as an argument - it works. As far as I understand it has something to do with the from some_package.foo import bar statement. If it's being executed before monkeypatching is happening then patching fails. But on the condensed test setup from the example above patching does not work in both cases. And why does it work in IPDB REPL after hitting a breakpoint?

Output: Named importation creates a new name for the object. If you then replace the old name for the object the new name is unaffected. Import the module and use module.bar instead. That will always use the current object. import module def func_under_test(): module.foo() def test_func(): monkeypatch.setattr(...) func_under_test


Instruction:
Reading named command arguments

Can I use argparse to read named command line arguments that do not need to be in a specific order? I browsed through the documentation but most of it focused on displaying content based on the arguments provided (such as --h). Right now, my script reads ordered, unnamed arguments: myscript.py foo-val bar-val using sys.argv: foo = sys.argv[1] bar = sys.argv[2] But I would like to change the input so that it is order agnostic by naming arguments: myscript.py --bar=bar-val --foo=foo-val

Output: You can use the Optional Arguments like so. With this program: #!/usr/bin/env python3 import argparse, sys parser=argparse.ArgumentParser() parser.add_argument("--bar", help="Do the bar option") parser.add_argument("--foo", help="Foo the program") args=parser.parse_args() print(f"Args: {args}\nCommand Line: {sys.argv}\nfoo: {args.foo}") print(f"Dict format: {vars(args)}") Make it executable: $ chmod +x prog.py Then if you call it with: $ ./prog.py --bar=bar-val --foo foo-val It prints: Args: Namespace(bar='bar-val', foo='foo-val') Command Line: ['./prog.py', '--bar=bar-val', '--foo', 'foo-val'] foo: foo-val Dict format: {'bar': 'bar-val', 'foo': 'foo-val'} Or, if the user wants help argparse builds that too: $ ./prog.py -h usage: prog.py [-h] [--bar BAR] [--foo FOO] options: -h, --help show this help message and exit --bar BAR Do the bar option --foo FOO Foo the program 2022-08-30: Updated to Python3 this answer...


Instruction:
Passing table name as a parameter in psycopg2

I have the following code, using pscyopg2: sql = 'select %s from %s where utctime > %s and utctime < %s order by utctime asc;' data = (dataItems, voyage, dateRangeLower, dateRangeUpper) rows = cur.mogrify(sql, data) This outputs: select 'waterTemp, airTemp, utctime' from 'ss2012_t02' where utctime > '2012-05-03T17:01:35+00:00'::timestamptz and utctime < '2012-05-01T17:01:35+00:00'::timestamptz order by utctime asc; When I execute this, it falls over - this is understandable, as the quotes around the table name are illegal. Is there a way to legally pass the table name as a parameter, or do I need to do a (explicitly warned against) string concatenation, ie: voyage = 'ss2012_t02' sql = 'select %s from ' + voyage + ' where utctime > %s and utctime < %s order by utctime asc;' Cheers for any insights.

Output: According to the official documentation: If you need to generate dynamically an SQL query (for instance choosing dynamically a table name) you can use the facilities provided by the psycopg2.sql module. The sql module is new in psycopg2 version 2.7. It has the following syntax: from psycopg2 import sql cur.execute( sql.SQL("insert into {table} values (%s, %s)") .format(table=sql.Identifier('my_table')), [10, 20]) More on: https://www.psycopg.org/docs/sql.html#module-usage [Update 2017-03-24: AsIs should NOT be used to represent table or fields names, the new sql module should be used instead: https://stackoverflow.com/a/42980069/5285608 ] Also, according to psycopg2 documentation: Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.


Instruction:
Get SQL query count during a Django shell session

Is there a way to print the number of raw SQL queries performed by the Django ORM during a Django shell session? This sort of information is already provided by the Django debug toolbar (e.g, 5 QUERIES in 5.83MS but it's not obvious how to get it from the shell.

Output: You can use connection.queries: >>> from django.conf import settings >>> settings.DEBUG = True >>> from django.db import connection >>> Model.objects.count() >>> print(len(connection.queries)) 1


Instruction:
Print Combining Strings and Numbers

To print strings and numbers in Python, is there any other way than doing something like: first = 10 second = 20 print "First number is %(first)d and second number is %(second)d" % {"first": first, "second":second}

Output: Using print function without parentheses works with older versions of Python but is no longer supported on Python3, so you have to put the arguments inside parentheses. However, there are workarounds, as mentioned in the answers to this question. Since the support for Python2 has ended in Jan 1st 2020, the answer has been modified to be compatible with Python3. You could do any of these (and there may be other ways): (1) print("First number is {} and second number is {}".format(first, second)) (1b) print("First number is {first} and number is {second}".format(first=first, second=second)) or (2) print('First number is', first, 'second number is', second) (Note: A space will be automatically added afterwards when separated from a comma) or (3) print('First number %d and second number is %d' % (first, second)) or (4) print('First number is ' + str(first) + ' second number is' + str(second)) Using format() (1/1b) is preferred where available.


Instruction:
Reset color cycle in Matplotlib

Say I have data about 3 trading strategies, each with and without transaction costs. I want to plot, on the same axes, the time series of each of the 6 variants (3 strategies * 2 trading costs). I would like the "with transaction cost" lines to be plotted with alpha=1 and linewidth=1 while I want the "no transaction costs" to be plotted with alpha=0.25 and linewidth=5. But I would like the color to be the same for both versions of each strategy. I would like something along the lines of: fig, ax = plt.subplots(1, 1, figsize=(10, 10)) for c in with_transaction_frame.columns: ax.plot(with_transaction_frame[c], label=c, alpha=1, linewidth=1) ****SOME MAGIC GOES HERE TO RESET THE COLOR CYCLE for c in no_transaction_frame.columns: ax.plot(no_transaction_frame[c], label=c, alpha=0.25, linewidth=5) ax.legend() What is the appropriate code to put on the indicated line to reset the color cycle so it is "back to the start" when the second loop is invoked?

Output: In Matplotlib <1.5.0, you can reset the colorcycle to the original with Axes.set_color_cycle. Looking at the code for this, there is a function to do the actual work: def set_color_cycle(self, clist=None): if clist is None: clist = rcParams['axes.color_cycle'] self.color_cycle = itertools.cycle(clist) And a method on the Axes which uses it: def set_color_cycle(self, clist): """ Set the color cycle for any future plot commands on this Axes. *clist* is a list of mpl color specifiers. """ self._get_lines.set_color_cycle(clist) self._get_patches_for_fill.set_color_cycle(clist) This basically means you can call the set_color_cycle with None as the only argument, and it will be replaced with the default cycle found in rcParams['axes.color_cycle']. I tried this with the following code and got the expected result: import matplotlib.pyplot as plt import numpy as np for i in range(3): plt.plot(np.arange(10) + i) # for Matplotlib version < 1.5 plt.gca().set_color_cycle(None) # for Matplotlib version >= 1.5 plt.gca().set_prop_cycle(None) for i in range(3): plt.plot(np.arange(10, 1, -1) + i) plt.show()


Instruction:
Determining how many times a substring occurs in a string in Python

I am trying to figure out how many times a string occurs in a string. For example: nStr = '000123000123' Say the string I want to find is 123. Obviously it occurs twice in nStr but I am having trouble implementing this logic into Python. What I have got at the moment: pattern = '123' count = a = 0 while pattern in nStr[a:]: a = nStr[a:].find(pattern)+1 count += 1 return count The answer it should return is 2. I'm stuck in an infinite loop at the moment. I was just made aware that count is a much better way to do it but out of curiosity, does anyone see a way to do it similar to what I have already got?

Output: Use str.count: >>> nStr = '000123000123' >>> nStr.count('123') 2 A working version of your code: nStr = '000123000123' pattern = '123' count = 0 flag = True start = 0 while flag: a = nStr.find(pattern, start) # find() returns -1 if the word is not found, #start i the starting index from the search starts(default value is 0) if a == -1: #if pattern not found set flag to False flag = False else: # if word is found increase count and set starting index to a+1 count += 1 start = a + 1 print(count)


Instruction:
What is %pylab?

I keep seeing people use %pylab in various code snippits, particularly with iPython. However, I cannot see where %pylab is mentioned anywhere in Learning Python (and the few other Python books I have) and am not really sure what it means. I'm sure the answer is simple, but can anyone enlighten me?

Output: %pylab is a magic function in ipython. Magic functions in ipython always begin with the percent sign (%) followed without any spaces by a small text string; in essence, ipython magic functions define shortcuts particularly useful for interactive work, e.g., to give you an idea of how magic functions work in python, a few of my favorites: to view cwd directory contents: %ls to run a script in ipython using an empty namespace, type space then a script name: %run to execute a code snippet (particularly for multi-line snippets which would usually cause an _IndentationError_ to be thrown): %paste When the %pylab magic function is entered at the IPython prompt, it triggers the import of various modules within Matplotlib. Which modules? well, the ones subsumed under the pylab interface. The awesome Matplotlib plotting library has two distinct interfaces: a pythonic one, and the original MATLAB-like one intended for plotting at the interactive prompt. The former is usually imported like so: from matplotlib import pyplot as PLT Indeed, pyplot has its own magic python magic function %pyplot Why two different interfaces? Matplotlib's original interface was pylab; only later was the pythonic interface added. Scripting and app development were not the primary uses cases for Matplotlib when the project began, plotting in the python shell was. Apparently John Hunter (Matplotlib's creator) wanted to include interactive plotting in python so he submitted a patch to Fernando Perez's (FP) IPython project. FP was a Ph.D student at the time and informed JH that he would not able to review the path for some time. As a result, JH created Matplotlib. The significance is that Matplotlib began as a shell-based plotting scheme. the pylab interface is indeed more suitable for interactive work: from pylab import * x, y = arange(10), cos(x/2) plot(x, y) show() and using the pyplot interface: from matplotlib import pyplot as PLT import numpy as NP x, y = NP.arange(10), NP.cos(x/2) fig = PLT.figure() ax1 = fig.add_subplot(111) ax1.plot(x, y) PLT.show()


Instruction:
setup.py and adding file to /bin/

I can't figure out how to make setup.py add a script to the the user's /bin or /usr/bin or whatever. E.g., I'd like to add a myscript.py to /usr/bin so that the user can call myscript.py from any directory.

Output: The Python documentation explains it under the installing scripts section. Scripts are files containing Python source code, intended to be started from the command line. setup(..., scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val'] ) As mentioned here, beside scripts, there is an entry_points mechanism, which is more cross-platform. With entry_points you connect a command line tool name with a function of your choice, whereas scripts could point to any file (e.g. a shell script).


Instruction:
What are the best practices for structuring a FastAPI project?

The problem that I want to solve related the project setup: Good names of directories so that their purpose is clear. Keeping all project files (including virtualenv) in one place, so I can easily copy, move, archive, remove the whole project, or estimate disk space usage. Creating multiple copies of some selected file sets such as entire application, repository, or virtualenv, while keeping a single copy of other files that I don't want to clone. Deploying the right set of files to the server simply by resyncing selected one dir. handling both frontend and backend nicely.

Output: Harsha already mentioned my project generator but I think it can be helpful for future readers to explain the ideas behind of it. If you are going to serve your frontend something like yarn or npm. You should not worry about the structure between them. With something like axios or the Javascript's fetch you can easily talk with your backend from anywhere. When it comes to structuring the backend, if you want to render templates with Jinja, you can have something that is close to MVC Pattern. your_project ├── __init__.py ├── main.py ├── core │ ├── models │ │ ├── database.py │ │ └── __init__.py │ ├── schemas │ │ ├── __init__.py │ │ └── schema.py │ └── settings.py ├── tests │ ├── __init__.py │ └── v1 │ ├── __init__.py │ └── test_v1.py └── v1 ├── api.py ├── endpoints │ ├── endpoint.py │ └── __init__.py └── __init__.py By using __init__ everywhere, we can access the variables from the all over the app, just like Django. Let's the folders into parts: Core models database.py schemas users.py something.py settings.py views (Add this if you are going to render templates) v1_views.py v2_views.py tests v1 v2 Models It is for your database models, by doing this you can import the same database session or object from v1 and v2. Schemas Schemas are your Pydantic models, we call it schemas because it is actually used for creating OpenAPI schemas since FastAPI is based on OpenAPI specification we use schemas everywhere, from Swagger generation to endpoint's expected request body. settings.py It is for Pydantic's Settings Management which is extremely useful, you can use the same variables without redeclaring it, to see how it could be useful for you check out our documentation for Settings and Environment Variables Views This is optional if you are going to render your frontend with Jinja, you can have something close to MVC pattern Core views v1_views.py v2_views.py It would look something like this if you want to add views. Tests It is good to have your tests inside your backend folder. APIs Create them independently by APIRouter, instead of gathering all your APIs inside one file. Notes You can use absolute import for all your importing since we are using __init__ everywhere, see Python's packaging docs. So assume you are trying to import v1's endpoint.py from v2, you can simply do from my_project.v1.endpoints.endpoint import something


Instruction:
Logging Error: Failed to initialize logging system. Log messages may be missing.?

Logging Error: Failed to initialize logging system. Log messages may be missing. If this issue persists, try setting IDEPreferLogStreaming=YES in the active scheme actions environment variables. Has anyone else encountered this message? Where is IDEPreferLogStreaming located? I don't know what any of this means. It's building my app successfully but then loading it like its a computer using floppy discs (crazy slow). Any ideas? I tried wiping my OS and reinstalling. I've reinstalled Xcode twice now. Nothing. A colleague of mine is working on the same SwifUI project with no issues.

Output: To find IDEPreferLogStreaming, you need to go to Product -> Scheme -> Edit Scheme and then add it as a new Environment Variable yourself. IDEPreferLogStreaming=YES For me it didnt solve the issue though --- [Edit: it works for me now as well. Probably I was to quick saying it doesnt. Thanks for your feedback.]


Instruction:
Difference between TypeVar('T', A, B) and TypeVar('T', bound=Union[A, B])

What's the difference between the following two TypeVars? from typing import TypeVar, Union class A: pass class B: pass T = TypeVar("T", A, B) T = TypeVar("T", bound=Union[A, B]) I believe that in Python 3.12 this is the difference between these two bounds class Foo[T: (A, B)]: ... class Foo[T: A | B]: ... Here's an example of something I don't get: this passes type checking... T = TypeVar("T", bound=Union[A, B]) class AA(A): pass class X(Generic[T]): pass class XA(X[A]): pass class XAA(X[AA]): pass ...but with T = TypeVar("T", A, B), it fails with error: Value of type variable "T" of "X" cannot be "AA" Related: this question on the difference between Union[A, B] and TypeVar("T", A, B).

Output: When you do T = TypeVar("T", bound=Union[A, B]), you are saying T can be bound to either Union[A, B] or any subtype of Union[A, B]. It's upper-bounded to the union. So for example, if you had a function of type def f(x: T) -> T, it would be legal to pass in values of any of the following types: Union[A, B] (or a union of any subtypes of A and B such as Union[A, BChild]) A (or any subtype of A) B (or any subtype of B) This is how generics behave in most programming languages: they let you impose a single upper bound. But when you do T = TypeVar("T", A, B), you are basically saying T must be either upper-bounded by A or upper-bounded by B. That is, instead of establishing a single upper-bound, you get to establish multiple! So this means while it would be legal to pass in values of either types A or B into f, it would not be legal to pass in Union[A, B] since the union is neither upper-bounded by A nor B. So for example, suppose you had a iterable that could contain either ints or strs. If you want this iterable to contain any arbitrary mixture of ints or strs, you only need a single upper-bound of a Union[int, str]. For example: from typing import TypeVar, Union, List, Iterable mix1: List[Union[int, str]] = [1, "a", 3] mix2: List[Union[int, str]] = [4, "x", "y"] all_ints = [1, 2, 3] all_strs = ["a", "b", "c"] T1 = TypeVar('T1', bound=Union[int, str]) def concat1(x: Iterable[T1], y: Iterable[T1]) -> List[T1]: out: List[T1] = [] out.extend(x) out.extend(y) return out # Type checks a1 = concat1(mix1, mix2) # Also type checks (though your type checker may need a hint to deduce # you really do want a union) a2: List[Union[int, str]] = concat1(all_ints, all_strs) # Also type checks a3 = concat1(all_strs, all_strs) In contrast, if you want to enforce that the function will accept either a list of all ints or all strs but never a mixture of either, you'll need multiple upper bounds. T2 = TypeVar('T2', int, str) def concat2(x: Iterable[T2], y: Iterable[T2]) -> List[T2]: out: List[T2] = [] out.extend(x) out.extend(y) return out # Does NOT type check b1 = concat2(mix1, mix2) # Also does NOT type check b2 = concat2(all_ints, all_strs) # But this type checks b3 = concat2(all_ints, all_ints)


Instruction:
Can I get PyCharm to suppress a particular warning on a single line?

PyCharm provides some helpful warnings on code style, conventions and logical gotchas. It also provides a notification if I try to commit code with warnings (or errors). Sometimes I consciously ignore these warnings for particular lines of code (for various reasons, typically to account for implementation details of third-party libraries). I want to suppress the warning, but just for that line (if the warning crops up on a different line where I'm not being deliberate, I want to know about it!) How can I do that in PyCharm? (Following a universal Python convention strongly preferable.)

Output: To suppress PyCharm code inspections for a particular line of code you can use the following construct: # noinspection INSPECTION_NAME your_line_of_code_to_suppress where the name of the inspection (INSPECTION_NAME above) you can take from the list of inspection names (they are pretty descriptive). To suppress pylint command line messages explicitly you have to use different comments/commands, as described here (pylint error names).


Instruction:
which one should I use: os.sep or os.path.sep?

They are same, but which one should I use? http://docs.python.org/library/os.html: os.sep The character used by the operating system to separate pathname components. This is '/' for POSIX and '\' for Windows. Note that knowing this is not sufficient to be able to parse or concatenate pathnames — use os.path.split() and os.path.join() — but it is occasionally useful. Also available via os.path.

Output: I'd use os.path.sep to make it very clear that it's the path separator… But consistency is more important, so if one is already being used, use that. Otherwise, pick one and use it all the time. Edit: Just to make sure you're not reinventing the wheel, though, the path module already has join, split, dirname, and basename functions… So you should rarely need to use os.sep: >>> os.path.join("foo", "bar", "baz") 'foo/bar/baz' >>> os.path.split(_) ('foo/bar', 'baz')


Instruction:
Django - FileField check if None

I have a model with an optional file field class MyModel(models.Model): name = models.CharField(max_length=50) sound = models.FileField(upload_to='audio/', blank=True) Let's put a value >>> test = MyModel(name='machin') >>> test.save() Why do I get that ? >>> test.sound <FieldFile: None> >>> test.sound is None False How can I check if there is a file set ?

Output: if test.sound.name: print "I have a sound file" else: print "no sound" Also, FileField's boolean value will be False when there's no file: bool(test.sound) == False when test.sound.name is falsy.


Instruction:
Why Pytorch officially use mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225] to normalize images?

In this page (https://pytorch.org/vision/stable/models.html), it says that "All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]". Shouldn't the usual mean and std of normalization be [0.5, 0.5, 0.5] and [0.5, 0.5, 0.5]? Why is it setting such strange values?

Output: Using the mean and std of Imagenet is a common practice. They are calculated based on millions of images. If you want to train from scratch on your own dataset, you can calculate the new mean and std. Otherwise, using the Imagenet pretrianed model with its own mean and std is recommended.