Instruction:
background function in Python

I've got a Python script that sometimes displays images to the user. The images can, at times, be quite large, and they are reused often. Displaying them is not critical, but displaying the message associated with them is. I've got a function that downloads the image needed and saves it locally. Right now it's run inline with the code that displays a message to the user, but that can sometimes take over 10 seconds for non-local images. Is there a way I could call this function when it's needed, but run it in the background while the code continues to execute? I would just use a default image until the correct one becomes available.

Output: Do something like this: def function_that_downloads(my_args): # do some long download here then inline, do something like this: import threading def my_inline_function(some_args): # do some stuff download_thread = threading.Thread(target=function_that_downloads, name="Downloader", args=some_args) download_thread.start() # continue doing stuff You may want to check if the thread has finished before going on to other things by calling download_thread.isAlive()


Instruction:
How to turn on line numbers in IDLE?

In the main shell of IDLE, errors always return a line number but the development environment doesn't even have line numbers. Is there anyway to turn on line numbers?

Output: Version 3.8 or newer: To show line numbers in the current window, go to Options and click Show Line Numbers. To show them automatically, go to Options > Configure IDLE > General and check the Show line numbers in new windows box. Version 3.7 or older: Unfortunately there is not an option to display line numbers in IDLE although there is an enhancement request open for this. However, there are a couple of ways to work around this: Under the edit menu there is a go to line option (there is a default shortcut of Alt+G for this). There is a display at the bottom right which tells you your current line number / position on the line:


Instruction:
Using mock patch to mock an instance method

I'm trying to mock something while testing a Django app using the imaginatively named Mock testing library. I can't seem to quite get it to work, I'm trying to do this: models.py from somelib import FooClass class Promotion(models.Model): foo = models.ForeignKey(FooClass) def bar(self): print "Do something I don't want!" test.py class ViewsDoSomething(TestCase): view = 'my_app.views.do_something' def test_enter_promotion(self): @patch.object(my_app.models.FooClass, 'bar') def fake_bar(self, mock_my_method): print "Do something I want!" return True self.client.get(reverse(view)) What am I doing wrong?

Output: To add onto Kit's answer, specifying a 3rd argument to patch.object() allows the mocked object/method to be specified. Otherwise, a default MagicMock object is used. def fake_bar(self): print "Do something I want!" return True @patch.object(my_app.models.FooClass, 'bar', fake_bar) def test_enter_promotion(self): self.client.get(reverse(view)) # Do something I want! Note that, if you specify the mocking object as a third argument, then the default MagicMock() is no longer passed into the patched object -- e.g. no longer: def test_enter_promotion(self, mock_method): but instead: def test_enter_promotion(self): https://docs.python.org/3/library/unittest.mock.html#patch-object


Instruction:
I used `__metaclass__` to set up `abc.ABCMeta` as the metaclass, but unimplemented `@abstractmethod`s still fail to raise an exception. Why?

I have read python docs about abstract base classes: From here: abc.abstractmethod(function) A decorator indicating abstract methods. Using this decorator requires that the class’s metaclass is ABCMeta or is derived from it. A class that has a metaclass derived from ABCMeta cannot be instantiated unless all of its abstract methods and properties are overridden. And here You can apply the @abstractmethod decorator to methods such as draw() that must be implemented; Python will then raise an exception for classes that don’t define the method. Note that the exception is only raised when you actually try to create an instance of a subclass lacking the method. I've used this code to test that out: import abc class AbstractClass(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod def abstractMethod(self): return class ConcreteClass(AbstractClass): def __init__(self): self.me = "me" c = ConcreteClass() c.abstractMethod() The code goes fine, so I don't get it. If I type c.abstractMethod I get: <bound method ConcreteClass.abstractMethod of <__main__.ConcreteClass object at 0x7f694da1c3d0>> What I'm missing here? ConcreteClass must implement the abstract methods, but I get no exception.

Output: Are you using python3 to run that code? If yes, you should know that declaring metaclass in python3 have changes you should do it like this instead: import abc class AbstractClass(metaclass=abc.ABCMeta): @abc.abstractmethod def abstractMethod(self): return The full code and the explanation behind the answer is: import abc class AbstractClass(metaclass=abc.ABCMeta): @abc.abstractmethod def abstractMethod(self): return class ConcreteClass(AbstractClass): def __init__(self): self.me = "me" # Will get a TypeError without the following two lines: # def abstractMethod(self): # return 0 c = ConcreteClass() c.abstractMethod() If abstractMethod is not defined for ConcreteClass, the following exception will be raised when running the above code: TypeError: Can't instantiate abstract class ConcreteClass with abstract methods abstractMethod


Instruction:
How can I print a single backslash?

When I write print('\') or print("\") or print("'\'"), Python doesn't print the backslash \ symbol. Instead it errors for the first two and prints '' for the third. What should I do to print a backslash? This question is about producing a string that has a single backslash in it. This is particularly tricky because it cannot be done with raw strings. For the related question about why such a string is represented with two backslashes, see Why do backslashes appear twice?. For including literal backslashes in other strings, see using backslash in python (not to escape).

Output: You need to escape your backslash by preceding it with, yes, another backslash: print("\\") The \ character is called an escape character, which interprets the character following it differently. For example, n by itself is simply a letter, but when you precede it with a backslash, it becomes \n, which is the newline character. As you can probably guess, \ also needs to be escaped so it doesn't function like an escape character. You have to... escape the escape, essentially. See the documentation for string literals.


Instruction:
What is the currently correct way to dynamically update plots in Jupyter/iPython?

In the answers to how to dynamically update a plot in a loop in ipython notebook (within one cell), an example is given of how to dynamically update a plot inside a Jupyter notebook within a Python loop. However, this works by destroying and re-creating the plot on every iteration, and a comment in one of the threads notes that this situation can be improved by using the new-ish %matplotlib nbagg magic, which provides an interactive figure embedded in the notebook, rather than a static image. However, this wonderful new nbagg feature seems to be completely undocumented as far as I can tell, and I'm unable to find an example of how to use it to dynamically update a plot. Thus my question is, how does one efficiently update an existing plot in a Jupyter/Python notebook, using the nbagg backend? Since dynamically updating plots in matplotlib is a tricky issue in general, a simple working example would be an enormous help. A pointer to any documentation on the topic would also be extremely helpful. To be clear what I'm asking for: what I want to do is to run some simulation code for a few iterations, then draw a plot of its current state, then run it for a few more iterations, then update the plot to reflect the current state, and so on. So the idea is to draw a plot and then, without any interaction from the user, update the data in the plot without destroying and re-creating the whole thing. Here is some slightly modified code from the answer to the linked question above, which achieves this by re-drawing the whole figure every time. I want to achieve the same result, but more efficiently using nbagg. %matplotlib inline import time import pylab as pl from IPython import display for i in range(10): pl.clf() pl.plot(pl.randn(100)) display.display(pl.gcf()) display.clear_output(wait=True) time.sleep(1.0)

Output: Here is an example that updates a plot in a loop. It updates the data in the figure and does not redraw the whole figure every time. It does block execution, though if you're interested in running a finite set of simulations and saving the results somewhere, it may not be a problem for you. The %matplotlib widget magic requires the ipympl Matplotlib Jupyter Extension package. You can install a working environment with pip install jupyter ipympl %matplotlib widget import numpy as np import matplotlib.pyplot as plt import time def pltsin(ax, colors=['b']): x = np.linspace(0,1,100) if ax.lines: for line in ax.lines: line.set_xdata(x) y = np.random.random(size=(100,1)) line.set_ydata(y) else: for color in colors: y = np.random.random(size=(100,1)) ax.plot(x, y, color) fig.canvas.draw() fig,ax = plt.subplots(1,1) ax.set_xlabel('X') ax.set_ylabel('Y') ax.set_xlim(0,1) ax.set_ylim(0,1) plt.show() # run this cell to dynamically update plot for f in range(5): pltsin(ax, ['b', 'r']) time.sleep(1) I put this up on nbviewer here, and here's a direct link to the gist


Instruction:
How does Lru_cache (from functools) Work?

Especially when using recursive code there are massive improvements with lru_cache. I do understand that a cache is a space that stores data that has to be served fast and saves the computer from recomputing. How does the Python lru_cache from functools work internally? I'm Looking for a specific answer, does it use dictionaries like the rest of Python? Does it only store the return value? I know that Python is heavily built on top of dictionaries, however, I couldn't find a specific answer to this question.

Output: The functools source code is available here: https://github.com/python/cpython/blob/master/Lib/functools.py lru_cache uses the _lru_cache_wrapper decorator (python decorator with arguments pattern) which has a cache dictionary in context in which it saves the return value of the function called (every decorated function will have its own cache dict). The dictionary key is generated with the _make_key function from the arguments. Added some bold comments below: # ACCORDING TO PASSED maxsize ARGUMENT _lru_cache_wrapper # DEFINES AND RETURNS ONE OF wrapper DECORATORS def _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo): # Constants shared by all lru cache instances: sentinel = object() # unique object used to signal cache misses cache = {} # RESULTS SAVES HERE cache_get = cache.get # bound method to lookup a key or return None # ... maxsize is None: def wrapper(*args, **kwds): # Simple caching without ordering or size limit nonlocal hits, misses key = make_key(args, kwds, typed) # BUILD A KEY FROM ARGUMENTS result = cache_get(key, sentinel) # TRYING TO GET PREVIOUS CALLS RESULT if result is not sentinel: # ALREADY CALLED WITH PASSED ARGS hits += 1 return result # RETURN SAVED RESULT # WITHOUT ACTUALLY CALLING FUNCTION misses += 1 result = user_function(*args, **kwds) # FUNCTION CALL - if cache[key] empty cache[key] = result # SAVE RESULT return result # ... return wrapper


Instruction:
How to save a list to a file and read it as a list type?

Say I have the list score = [1,2,3,4,5] and it gets changed while my program is running. How could I save it to a file so that next time the program is run I can access the changed list as a list type? I have tried: score=[1,2,3,4,5] with open("file.txt", 'w') as f: for s in score: f.write(str(s) + '\n') with open("file.txt", 'r') as f: score = [line.rstrip('\n') for line in f] print(score) But this results in the elements in the list being strings not integers.

Output: I decided I didn't want to use a pickle because I wanted to be able to open the text file and change its contents easily during testing. Therefore, I did this: score = [1,2,3,4,5] with open("file.txt", "w") as f: for s in score: f.write(str(s) +"\n") score = [] with open("file.txt", "r") as f: for line in f: score.append(int(line.strip())) So the items in the file are read as integers, despite being stored to the file as strings.


Instruction:
Find common substring between two strings

I'd like to compare 2 strings and keep the matched, splitting off where the comparison fails. So if I have 2 strings: string1 = "apples" string2 = "appleses" answer = "apples" Another example, as the string could have more than one word: string1 = "apple pie available" string2 = "apple pies" answer = "apple pie" I'm sure there is a simple Python way of doing this but I can't work it out, any help and explanation appreciated.

Output: Its called Longest Common Substring problem. Here I present a simple, easy to understand but inefficient solution. It will take a long time to produce correct output for large strings, as the complexity of this algorithm is O(N^2). def longestSubstringFinder(string1, string2): answer = "" len1, len2 = len(string1), len(string2) for i in range(len1): match = "" for j in range(len2): if (i + j < len1 and string1[i + j] == string2[j]): match += string2[j] else: if (len(match) > len(answer)): answer = match match = "" return answer print(longestSubstringFinder("apple pie available", "apple pies")) print(longestSubstringFinder("apples", "appleses")) print(longestSubstringFinder("bapples", "cappleses")) Output apple pie apples apples


Instruction:
unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed with ChromeDriver Selenium

I'm using InstaPy which use Python and Selenium. I start the script per Cron and from time to time it crashes. So it'r really irregular, sometimes it runs well through. I'v posted on GitHub Repo as well already but didn't get an answer there, so i'm asking here now if someone has an idea why. It's a digital ocean ubuntu server and i'm using it on headless mode. The driver version are visible on the log. here are error messages: ERROR [2018-12-10 09:53:54] [user] Error occurred while deleting cookies from web browser! b'Message: invalid session id\n (Driver info: chromedriver=2.44.609551 (5d576e9a44fe4c5b6a07e568f1ebc753f1214634),platform=Linux 4.15.0-42-generic x86_64)\n' Traceback (most recent call last): File "/root/InstaPy/instapy/util.py", line 1410, in smart_run yield File "./my_config.py", line 43, in <module> session.follow_user_followers(['xxxx','xxxx','xxxx','xxxx'], amount=100, randomize=True, interact=True) File "/root/InstaPy/instapy/instapy.py", line 2907, in follow_user_followers self.logfolder) File "/root/InstaPy/instapy/unfollow_util.py", line 883, in get_given_user_followers channel, jumps, logger, logfolder) File "/root/InstaPy/instapy/unfollow_util.py", line 722, in get_users_through_dialog person_list = dialog_username_extractor(buttons) File "/root/InstaPy/instapy/unfollow_util.py", line 747, in dialog_username_extractor person_list.append(person.find_element_by_xpath("../../../*") File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py", line 351, in find_element_by_xpath return self.find_element(by=By.XPATH, value=xpath) File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py", line 659, in find_element {"using": by, "value": value})['value'] File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute return self._parent.execute(command, params) File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed (Session info: headless chrome=70.0.3538.110) (Driver info: chromedriver=2.44.609551 (5d576e9a44fe4c5b6a07e568f1ebc753f1214634),platform=Linux 4.15.0-42-generic x86_64) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/InstaPy/instapy/instapy.py", line 3845, in end self.browser.delete_all_cookies() File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 878, in delete_all_cookies self.execute(Command.DELETE_ALL_COOKIES) File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: chrome not reachable (Session info: headless chrome=71.0.3578.80) (Driver info: chromedriver=2.44.609551 (5d576e9a44fe4c5b6a07e568f1ebc753f1214634),platform=Linux 4.15.0-42-generic x86_64) Any idea what the reason could be and how to solve it? Thanks for the inputs. And the guys from http://treestones.ch/ helped me out.

Output: Though you see the error as: Error occurred while deleting cookies from web browser! b'Message: invalid session id\n (Driver info: chromedriver=2.44.609551 (5d576e9a44fe4c5b6a07e568f1ebc753f1214634),platform=Linux 4.15.0-42-generic x86_64)\n' The main exception is: selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crashed Your code trials would have given us some clues what going wrong. Solution There are diverse solution to this issue. However as per UnknownError: session deleted because of page crash from tab crashed this issue can be solved by either of the following solutions: Add the following chrome_options: chrome_options.add_argument('--no-sandbox') Chrome seem to crash in Docker containers on certain pages due to too small /dev/shm. So you may have to fix the small /dev/shm size. An example: sudo mount -t tmpfs -o rw,nosuid,nodev,noexec,relatime,size=512M tmpfs /dev/shm It also works if you use -v /dev/shm:/dev/shm option to share host /dev/shm Another way to make it work would be to add the chrome_options as --disable-dev-shm-usage. This will force Chrome to use the /tmp directory instead. This may slow down the execution though since disk will be used instead of memory. chrome_options.add_argument('--disable-dev-shm-usage') from tab crashed from tab crashed was WIP(Work In Progress) with the Chromium Team for quite some time now which relates to Linux attempting to always use /dev/shm for non-executable memory. Here are the references : Linux: Chrome/Chromium SIGBUS/Aw, Snap! on small /dev/shm Chrome crashes/fails to load when /dev/shm is too small, and location can't be overridden As per Comment61#Issue 736452 the fix seems to be have landed with Chrome v65.0.3299.6 Reference You can find a couple of relevant discussions in: org.openqa.selenium.SessionNotCreatedException: session not created exception from tab crashed error when executing from Jenkins CI server


Instruction:
Matplotlib cannot find basic fonts

I am using matplotlib version 2.0.0 on Python 3 in a miniconda virtual environment. I am working on a unix scientific computing cluster where I don't have root privileges. I am generally executing python code through an ipython notebook. If I do a basic command such as: import matplotlib.pyplot as plt plt.scatter([1,5], [1,5]) I get an error message: path_to_miniconda/miniconda3/envs/conda34/lib/python3.4/site- packages/matplotlib/font_manager.py:1297: UserWarning: findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans (prop.get_family(), self.defaultFamily[fontext])) I would like to be able to use a Times New Roman font but even after deleting the Font cache file (fontList.py3k.cache) which I find from here: import matplotlib as mpl fm = mpl.font_manager fm.get_cachedir() The commands: mpl.rcParams['font.family'] = ['serif'] mpl.rcParams['font.serif'] = ['Times New Roman'] have no effect, I get the same error as above. The true type fonts directory: path_to_miniconda/miniconda3/envs/conda34/lib/python3.4/site-packages/matplotlib/mpl-data/fonts/ttf/ only has 40 fonts in it of the type: DejaVuSerif,DejaVuSans,STIX,cmb, cmt, cmy Any idea what could be going on and how I can add additional fonts? Thanks!

Output: To get it to work, I had to combine the two current top answers. Here's what worked for me: sudo apt install msttcorefonts -qq rm ~/.cache/matplotlib -rf # remove cache


Instruction:
What is `id()` function used for in Python?

I read the Python 2 docs and noticed the id() function: Return the “identity” of an object. This is an integer (or long integer) which is guaranteed to be unique and constant for this object during its lifetime. Two objects with non-overlapping lifetimes may have the same id() value. CPython implementation detail: This is the address of the object in memory. So, I experimented by using id() with a list: >>> list = [1,2,3] >>> id(list[0]) 31186196 >>> id(list[1]) 31907092 // increased by 896 >>> id(list[2]) 31907080 // decreased by 12 What is the integer returned from the function? Is it synonymous to memory addresses in C? If so, why doesn't the integer correspond to the size of the data type? When is id() used in practice?

Output: Your post asks several questions: What is the number returned from the function? It is "an integer (or long integer) which is guaranteed to be unique and constant for this object during its lifetime." (Python Standard Library - Built-in Functions) A unique number. Nothing more, and nothing less. Think of it as a social-security number or employee id number for Python objects. Is it the same with memory addresses in C? Conceptually, yes, in that they are both guaranteed to be unique in their universe during their lifetime. And in one particular implementation of Python, it actually is the memory address of the corresponding C object. If yes, why doesn't the number increase instantly by the size of the data type (I assume that it would be int)? Because a list is not an array, and a list element is a reference, not an object. When do we really use id( ) function? Aside from debugging, whenever you only care about the identity of objects and not their value equality (__eq__ or other built-in value equality), and when the reference itself is not desirable or possible to use, e.g. using arbitrary object identity as a dict key, or using integers instead of references for optimization purposes. You should not use it to test if two references are the same by comparing their ids, as the is operator has always been the recommended way of doing that.


Instruction:
How to dynamically update a plot in a loop in IPython notebook (within one cell)

Environment: Python 2.7, Matplotlib 1.3, IPython notebook 1.1, Linux, and Chrome. The code is in one single input cell, using --pylab=inline. I want to use IPython notebook and Pandas to consume a stream and dynamically update a plot every five seconds. When I just use a print statement to print the data in text format, it works perfectly fine: the output cell just keeps printing data and adding new rows. But when I try to plot the data (and then update it in a loop), the plot never shows up in the output cell. But if I remove the loop, and just plot it once, it works fine. Then I did some simple test: i = pd.date_range('2013-1-1',periods=100,freq='s') while True: plot(pd.Series(data=np.random.randn(100), index=i)) #pd.Series(data=np.random.randn(100), index=i).plot() also tried this one time.sleep(5) The output will not show anything until I manually interrupt the process (Ctrl + M + I). And after I interrupt it, the plot shows correctly as multiple overlapped lines. But what I really want is a plot that shows up and gets updated every five seconds (or whenever the plot() function gets called, just like what print statement outputs I mentioned above, which works well). Only showing the final chart after the cell is completely done is not what I want. I even tried to explicitly add the draw() function after each plot(), etc. None of them works. How can I dynamically update a plot by a for/while loop within one cell in IPython notebook?

Output: Use the IPython.display module: %matplotlib inline import time import pylab as pl from IPython import display for i in range(10): pl.plot(pl.randn(100)) display.clear_output(wait=True) display.display(pl.gcf()) time.sleep(1.0)


Instruction:
Nested For Loops Using List Comprehension

If I had two strings, 'abc' and 'def', I could get all combinations of them using two for loops: for j in s1: for k in s2: print(j, k) However, I would like to be able to do this using list comprehension. I've tried many ways, but have never managed to get it. Does anyone know how to do this?

Output: lst = [j + k for j in s1 for k in s2] or lst = [(j, k) for j in s1 for k in s2] if you want tuples. Like in the question, for j... is the outer loop, for k... is the inner loop. Essentially, you can have as many independent 'for x in y' clauses as you want in a list comprehension just by sticking one after the other. To make it more readable, use multiple lines: lst = [ j + k # result for j in s1 # for loop for k in s2 # for loop # condition ]


Instruction:
Is 'file' a keyword in python?

Is file a keyword in python? I've seen some code using the keyword file just fine, while others have suggested not to use it and my editor is color coding it as a keyword.

Output: No, file is not a keyword: >>> import keyword >>> keyword.iskeyword('file') False The name is not present in Python 3. In Python 2, file is a built-in: >>> import __builtin__, sys >>> hasattr(__builtin__, 'file') True >>> sys.version_info[:2] (2, 7) It can be seen as an alias for open(), but it was removed in Python 3, where the new io framework replaced it. Technically, it is the type of object returned by the Python 2 open() function.


Instruction:
How do you determine which backend is being used by matplotlib?

Either interactively, such as from within an Ipython session, or from within a script, how can you determine which backend is being used by matplotlib?

Output: Use the get_backend() function to obtain a string denoting which backend is in use: >>> import matplotlib >>> matplotlib.get_backend() 'TkAgg'


Instruction:
How to concatenate two layers in keras?

I have an example of a neural network with two layers. The first layer takes two arguments and has one output. The second should take one argument as result of the first layer and one additional argument. It should looks like this: x1 x2 x3 \ / / y1 / \ / y2 So, I'd created a model with two layers and tried to merge them but it returns an error: The first layer in a Sequential model must get an "input_shape" or "batch_input_shape" argument. on the line result.add(merged). Model: first = Sequential() first.add(Dense(1, input_shape=(2,), activation='sigmoid')) second = Sequential() second.add(Dense(1, input_shape=(1,), activation='sigmoid')) result = Sequential() merged = Concatenate([first, second]) ada_grad = Adagrad(lr=0.1, epsilon=1e-08, decay=0.0) result.add(merged) result.compile(optimizer=ada_grad, loss=_loss_tensor, metrics=['accuracy'])

Output: You're getting the error because result defined as Sequential() is just a container for the model and you have not defined an input for it. Given what you're trying to build set result to take the third input x3. first = Sequential() first.add(Dense(1, input_shape=(2,), activation='sigmoid')) second = Sequential() second.add(Dense(1, input_shape=(1,), activation='sigmoid')) third = Sequential() # of course you must provide the input to result which will be your x3 third.add(Dense(1, input_shape=(1,), activation='sigmoid')) # lets say you add a few more layers to first and second. # concatenate them merged = Concatenate([first, second]) # then concatenate the two outputs result = Concatenate([merged, third]) ada_grad = Adagrad(lr=0.1, epsilon=1e-08, decay=0.0) result.compile(optimizer=ada_grad, loss='binary_crossentropy', metrics=['accuracy']) However, my preferred way of building a model that has this type of input structure would be to use the functional api. Here is an implementation of your requirements to get you started: from keras.models import Model from keras.layers import Concatenate, Dense, LSTM, Input, concatenate from keras.optimizers import Adagrad first_input = Input(shape=(2, )) first_dense = Dense(1, )(first_input) second_input = Input(shape=(2, )) second_dense = Dense(1, )(second_input) merge_one = concatenate([first_dense, second_dense]) third_input = Input(shape=(1, )) merge_two = concatenate([merge_one, third_input]) model = Model(inputs=[first_input, second_input, third_input], outputs=merge_two) ada_grad = Adagrad(lr=0.1, epsilon=1e-08, decay=0.0) model.compile(optimizer=ada_grad, loss='binary_crossentropy', metrics=['accuracy']) To answer the question in the comments: How are result and merged connected? Assuming you mean how are they concatenated. Concatenation works like this: a b c a b c g h i a b c g h i d e f j k l d e f j k l i.e rows are just joined. Now, x1 is input to first, x2 is input into second and x3 input into third.


Instruction:
Passing categorical data to Sklearn Decision Tree

There are several posts about how to encode categorical data to Sklearn Decision trees, but from Sklearn documentation, we got these Some advantages of decision trees are: (...) Able to handle both numerical and categorical data. Other techniques are usually specialized in analyzing datasets that have only one type of variable. See the algorithms for more information. But running the following script import pandas as pd from sklearn.tree import DecisionTreeClassifier data = pd.DataFrame() data['A'] = ['a','a','b','a'] data['B'] = ['b','b','a','b'] data['C'] = [0, 0, 1, 0] data['Class'] = ['n','n','y','n'] tree = DecisionTreeClassifier() tree.fit(data[['A','B','C']], data['Class']) outputs the following error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/site-packages/sklearn/tree/tree.py", line 154, in fit X = check_array(X, dtype=DTYPE, accept_sparse="csc") File "/usr/local/lib/python2.7/site-packages/sklearn/utils/validation.py", line 377, in check_array array = np.array(array, dtype=dtype, order=order, copy=copy) ValueError: could not convert string to float: b I know that in R it is possible to pass categorical data, with Sklearn, is it possible?

Output: I would prefer to use tools provided by Scikit-Learn for this purpose. The main reason for doing so is that they can be easily integrated in a Pipeline. Scikit-Learn itself provides very good classes to handle categorical data. Instead of writing your custom function, you should use LabelEncoder which is specially designed for this purpose. Refer to the following code from the documentation: from sklearn import preprocessing le = preprocessing.LabelEncoder() le.fit(["paris", "paris", "tokyo", "amsterdam"]) le.transform(["tokyo", "tokyo", "paris"]) This automatically encodes them into numbers for your machine learning algorithms. Now this also supports going back to strings from integers. You can do this by simply calling inverse_transform as follows: list(le.inverse_transform([2, 2, 1])) This would return ['tokyo', 'tokyo', 'paris']. Also note that for many other classifiers, apart from decision trees, such as logistic regression or SVM, you would like to encode your categorical variables using One-Hot encoding. Scikit-learn supports this as well through the OneHotEncoder class.


Instruction:
Python threading. How do I lock a thread?

I'm trying to understand the basics of threading and concurrency. I want a simple case where two threads repeatedly try to access one shared resource. The code: import threading class Thread(threading.Thread): def __init__(self, t, *args): threading.Thread.__init__(self, target=t, args=args) self.start() count = 0 lock = threading.Lock() def increment(): global count lock.acquire() try: count += 1 finally: lock.release() def bye(): while True: increment() def hello_there(): while True: increment() def main(): hello = Thread(hello_there) goodbye = Thread(bye) while True: print count if __name__ == '__main__': main() So, I have two threads, both trying to increment the counter. I thought that if thread 'A' called increment(), the lock would be established, preventing 'B' from accessing until 'A' has released. Running the makes it clear that this is not the case. You get all of the random data race-ish increments. How exactly is the lock object used? Additionally, I've tried putting the locks inside of the thread functions, but still no luck.

Output: You can see that your locks are pretty much working as you are using them, if you slow down the process and make them block a bit more. You had the right idea, where you surround critical pieces of code with the lock. Here is a small adjustment to your example to show you how each waits on the other to release the lock. import threading import time import inspect class Thread(threading.Thread): def __init__(self, t, *args): threading.Thread.__init__(self, target=t, args=args) self.start() count = 0 lock = threading.Lock() def incre(): global count caller = inspect.getouterframes(inspect.currentframe())[1][3] print "Inside %s()" % caller print "Acquiring lock" with lock: print "Lock Acquired" count += 1 time.sleep(2) def bye(): while count < 5: incre() def hello_there(): while count < 5: incre() def main(): hello = Thread(hello_there) goodbye = Thread(bye) if __name__ == '__main__': main() Sample output: ... Inside hello_there() Acquiring lock Lock Acquired Inside bye() Acquiring lock Lock Acquired ...


Instruction:
numpy: formal definition of "array_like" objects?

In numpy, the constructors of many objects accept an "array_like" as first argument. Is there a definition of a such object, either as an abstract meta class, or documentation of the methods is should contain??

Output: It turns out almost anything is technically an array-like. "Array-like" is more of a statement of how the input will be interpreted than a restriction on what the input can be; if a parameter is documented as array-like, NumPy will try to interpret it as an array. There is no formal definition of array-like beyond the nearly tautological one -- an array-like is any Python object that np.array can convert to an ndarray. To go beyond this, you'd need to study the source code. NPY_NO_EXPORT PyObject * PyArray_FromAny(PyObject *op, PyArray_Descr *newtype, int min_depth, int max_depth, int flags, PyObject *context) { /* * This is the main code to make a NumPy array from a Python * Object. It is called from many different places. */ PyArrayObject *arr = NULL, *ret; PyArray_Descr *dtype = NULL; int ndim = 0; npy_intp dims[NPY_MAXDIMS]; /* Get either the array or its parameters if it isn't an array */ if (PyArray_GetArrayParamsFromObject(op, newtype, 0, &dtype, &ndim, dims, &arr, context) < 0) { Py_XDECREF(newtype); return NULL; } ... Particularly interesting is PyArray_GetArrayParamsFromObject, whose comments enumerate the types of objects np.array expects: NPY_NO_EXPORT int PyArray_GetArrayParamsFromObject(PyObject *op, PyArray_Descr *requested_dtype, npy_bool writeable, PyArray_Descr **out_dtype, int *out_ndim, npy_intp *out_dims, PyArrayObject **out_arr, PyObject *context) { PyObject *tmp; /* If op is an array */ /* If op is a NumPy scalar */ /* If op is a Python scalar */ /* If op supports the PEP 3118 buffer interface */ /* If op supports the __array_struct__ or __array_interface__ interface */ /* * If op supplies the __array__ function. * The documentation says this should produce a copy, so * we skip this method if writeable is true, because the intent * of writeable is to modify the operand. * XXX: If the implementation is wrong, and/or if actual * usage requires this behave differently, * this should be changed! */ /* Try to treat op as a list of lists */ /* Anything can be viewed as an object, unless it needs to be writeable */ } So by studying the source code we can conclude an array-like is a NumPy array, or a NumPy scalar, or a Python scalar, or any object which supports the PEP 3118 buffer interface, or any object that supports the __array_struct__ or __array_interface__ interface, or any object that supplies the __array__ function, or any object that can be treated as a list of lists, or anything! If it doesn't fall under one of the other cases, it'll be treated as a 0-dimensional array of object dtype.


Instruction:
virtualenv in PowerShell?

There seems to be a problem when virtualenv is used in PowerShell. When I try to activate my environment in PowerShell like... env/scripts/activate .. nothing happens. (the shell prompt should have changed as well as the PATH env. variable .) I guess the problem is that PowerShell spawns a new cmd. process just for running the activate.bat thus rendering the changes activate.bat does to the shell dead after it completes. Do you have any workarounds for the issue? (I'm sticking with cmd.exe for now)

Output: My original answer is now outdated. Now, just use activate.ps1 (rather than activate.bat) to activate from a Powershell environment. Original answer: Here's a post that contains a Powershell script that allows you to run batch files that persistently modify their environment variables. The script propagates any environment variable changes back to the calling PowerShell environment.


Instruction:
Interactively validating Entry widget content in tkinter

What is the recommended technique for interactively validating content in a tkinter Entry widget? I've read the posts about using validate=True and validatecommand=command, and it appears that these features are limited by the fact that they get cleared if the validatecommand command updates the Entry widget's value. Given this behavior, should we bind on the KeyPress, Cut, and Paste events and monitor/update our Entry widget's value through these events? (And other related events that I might have missed?) Or should we forget interactive validation altogether and only validate on FocusOut events?

Output: The correct answer is, use the validatecommand attribute of the widget. Unfortunately this feature is severely under-documented in the Tkinter world, though it is quite sufficiently documented in the Tk world. Even though it's not documented well, it has everything you need to do validation without resorting to bindings or tracing variables, or modifying the widget from within the validation procedure. The trick is to know that you can have Tkinter pass in special values to your validate command. These values give you all the information you need to know to decide on whether the data is valid or not: the value prior to the edit, the value after the edit if the edit is valid, and several other bits of information. To use these, though, you need to do a little voodoo to get this information passed to your validate command. Note: it's important that the validation command returns either True or False. Anything else will cause the validation to be turned off for the widget. Here's an example that only allows lowercase. It also prints the values of all of the special values for illustrative purposes. They aren't all necessary; you rarely need more than one or two. import tkinter as tk # python 3.x # import Tkinter as tk # python 2.x class Example(tk.Frame): def __init__(self, parent): tk.Frame.__init__(self, parent) # valid percent substitutions (from the Tk entry man page) # note: you only have to register the ones you need; this # example registers them all for illustrative purposes # # %d = Type of action (1=insert, 0=delete, -1 for others) # %i = index of char string to be inserted/deleted, or -1 # %P = value of the entry if the edit is allowed # %s = value of entry prior to editing # %S = the text string being inserted or deleted, if any # %v = the type of validation that is currently set # %V = the type of validation that triggered the callback # (key, focusin, focusout, forced) # %W = the tk name of the widget vcmd = (self.register(self.onValidate), '%d', '%i', '%P', '%s', '%S', '%v', '%V', '%W') self.entry = tk.Entry(self, validate="key", validatecommand=vcmd) self.text = tk.Text(self, height=10, width=40) self.entry.pack(side="top", fill="x") self.text.pack(side="bottom", fill="both", expand=True) def onValidate(self, d, i, P, s, S, v, V, W): self.text.delete("1.0", "end") self.text.insert("end","OnValidate:\n") self.text.insert("end","d='%s'\n" % d) self.text.insert("end","i='%s'\n" % i) self.text.insert("end","P='%s'\n" % P) self.text.insert("end","s='%s'\n" % s) self.text.insert("end","S='%s'\n" % S) self.text.insert("end","v='%s'\n" % v) self.text.insert("end","V='%s'\n" % V) self.text.insert("end","W='%s'\n" % W) # Disallow anything but lowercase letters if S == S.lower(): return True else: self.bell() return False if __name__ == "__main__": root = tk.Tk() Example(root).pack(fill="both", expand=True) root.mainloop() For more information about what happens under the hood when you call the register method, see Why is calling register() required for tkinter input validation? For the canonical documentation see the Validation section of the Tcl/Tk Entry man page


Instruction:
Django check for any exists for a query

In django how to check whether any entry exists for a query sc=scorm.objects.filter(Header__id=qp.id) This was how it was done in php if(mysql_num_rows($resultn)) { // True condition } else { // False condition }

Output: You can use exists(): if scorm.objects.filter(Header__id=qp.id).exists(): .... Returns True if the QuerySet contains any results, and False if not. This tries to perform the query in the simplest and fastest way possible, but it does execute nearly the same query as a normal QuerySet query. Older versions: (<1.2) Use count(): sc=scorm.objects.filter(Header__id=qp.id) if sc.count() > 0: ... The advantage over e.g. len() is, that the QuerySet is not yet evaluated: count() performs a SELECT COUNT(*) behind the scenes, so you should always use count() rather than loading all of the record into Python objects and calling len() on the result. Having this in mind, When QuerySets are evaluated can be worth reading. If you use get(), e.g. scorm.objects.get(pk=someid), and the object does not exists, an ObjectDoesNotExist exception is raised: from django.core.exceptions import ObjectDoesNotExist try: sc = scorm.objects.get(pk=someid) except ObjectDoesNotExist: print ...


Instruction:
Generate password in Python

I'd like to generate some alphanumeric passwords in Python. Some possible ways are: import string from random import sample, choice chars = string.ascii_letters + string.digits length = 8 ''.join(sample(chars, length)) # first way ''.join(choice(chars) for i in range(length)) # second way But I don't the first way because only unique chars are selected and you can't generate passwords where length > len(chars) and I don't like the second way because we have an unused i variable. Are there any other good options?

Output: On Python 3.6+ you should use the secrets module to generate cryptographically safe passwords. Adapted from the documentation: import secrets import string alphabet = string.ascii_letters + string.digits password = ''.join(secrets.choice(alphabet) for i in range(20)) # for a 20-character password For more information on recipes and best practices, see this section on recipes in the Python documentation. You can also consider adding string.punctuation.


Instruction:
How to run Pip commands from CMD

As I understand, Python 2.7.9 comes with Pip installed, however when I try to execute a Pip command from CMD (Windows) I get the following error: 'pip' is not recognized as an internal or external command, operable program or batch file. When I type python I do get the following, which suggests it has been installed correctly: Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. I did however need to add some environmental variables to get the python part working on CMD: Add to the environment variable PATH: "C:\Python27\" Define the system variable PYTHONPATH: "C:\Python27\" I cannot find a Pip folder within the Python directory, however there is a folder called "ensurepip" in C:\Python27\Lib\. Does anybody know how can I get Pip commands to start working in CMD?

Output: Little side note for anyone new to Python who didn't figure it out by theirself: this should be automatic when installing Python, but just in case, note that to run Python using the python command in Windows' CMD you must first add it to the PATH environment variable, as explained here. If you have the Python launcher installed, in such case instead of typing python in the console you can type py. Whether or not one or both commands are available depends on the choices you made during installation. To execute Pip, first of all make sure you have it installed, so type in your CMD: > python >>> import pip >>> The above should proceed with no error. Otherwise, if this fails, you can look here to see how to install it. Now that you are sure you've got Pip, you can run it from CMD with Python using the -m (module) parameter, like this: > python -m pip <command> <args> Where <command> is any Pip command you want to run, and <args> are its relative arguments, separated by spaces. For example, to install a package: > python -m pip install <package-name>


Instruction:
ImportError: No module named 'google'

I installed Python 3.5. I ran the pip install google command and verified the modules. Google was present. I installed Anaconda 3.5 and tried to run z sample code. But I'm getting the import error. Please find the screen shot attached. What am I missing? Do I have to link my Spyder to Python installation directory in some way? Why is Spyder unable to google module? My Python installation directory: C:\Users\XXX\AppData\Local\Programs\Python\Python35 My scenario is a bit different and I could not find a solution from similar posts here.

Output: I figured out the solution: I had to delete my Anaconda and Python installations Re-install Anaconda only Open the Anaconda prompt and point it to Anaconda/Scripts Run pip install google Run the sample code now from Spyder. No more errors.


Instruction:
How do I go straight to template, in Django's urls.py?

Instead of going to views.py, I want it to go to to a template, robots.txt.

Output: Django 2+ Note: is valid still as of Django 4+ Use the class based generic views but register with the django 2.0+ pattern. from django.urls import path from django.views.generic import TemplateView urlpatterns = [ path('foo/', TemplateView.as_view(template_name='foo.html')) ] https://docs.djangoproject.com/en/4.1/topics/class-based-views/#usage-in-your-urlconf Django 1.5+ Use the class based generic views. from django.views.generic import TemplateView urlpatterns = patterns('', (r'^foo/$', TemplateView.as_view(template_name='foo.html')), ) #Django <= 1.4 Docs: https://docs.djangoproject.com/en/1.4/ref/generic-views/#django-views-generic-simple-direct-to-template urlpatterns = patterns('django.views.generic.simple', (r'^foo/$', 'direct_to_template', {'template': 'foo_index.html'}), (r'^foo/(?P<id>\d+)/$', 'direct_to_template', {'template': 'foo_detail.html'}), )


Instruction:
How to calculate the inverse of the normal cumulative distribution function in python?

How do I calculate the inverse of the cumulative distribution function (CDF) of the normal distribution in Python? Which library should I use? Possibly scipy?

Output: NORMSINV (mentioned in a comment) is the inverse of the CDF of the standard normal distribution. Using scipy, you can compute this with the ppf method of the scipy.stats.norm object. The acronym ppf stands for percent point function, which is another name for the quantile function. In [20]: from scipy.stats import norm In [21]: norm.ppf(0.95) Out[21]: 1.6448536269514722 Check that it is the inverse of the CDF: In [34]: norm.cdf(norm.ppf(0.95)) Out[34]: 0.94999999999999996 By default, norm.ppf uses mean=0 and stddev=1, which is the "standard" normal distribution. You can use a different mean and standard deviation by specifying the loc and scale arguments, respectively. In [35]: norm.ppf(0.95, loc=10, scale=2) Out[35]: 13.289707253902945 If you look at the source code for scipy.stats.norm, you'll find that the ppf method ultimately calls scipy.special.ndtri. So to compute the inverse of the CDF of the standard normal distribution, you could use that function directly: In [43]: from scipy.special import ndtri In [44]: ndtri(0.95) Out[44]: 1.6448536269514722 ndtri is much faster than norm.ppf: In [46]: %timeit norm.ppf(0.95) 240 µs ± 1.75 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) In [47]: %timeit ndtri(0.95) 1.47 µs ± 1.3 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)


Instruction:
Pandas: rolling mean by time interval

I've got a bunch of polling data; I want to compute a Pandas rolling mean to get an estimate for each day based on a three-day window. According to this question, the rolling_* functions compute the window based on a specified number of values, and not a specific datetime range. How do I implement this functionality? Sample input data: polls_subset.tail(20) Out[185]: favorable unfavorable other enddate 2012-10-25 0.48 0.49 0.03 2012-10-25 0.51 0.48 0.02 2012-10-27 0.51 0.47 0.02 2012-10-26 0.56 0.40 0.04 2012-10-28 0.48 0.49 0.04 2012-10-28 0.46 0.46 0.09 2012-10-28 0.48 0.49 0.03 2012-10-28 0.49 0.48 0.03 2012-10-30 0.53 0.45 0.02 2012-11-01 0.49 0.49 0.03 2012-11-01 0.47 0.47 0.05 2012-11-01 0.51 0.45 0.04 2012-11-03 0.49 0.45 0.06 2012-11-04 0.53 0.39 0.00 2012-11-04 0.47 0.44 0.08 2012-11-04 0.49 0.48 0.03 2012-11-04 0.52 0.46 0.01 2012-11-04 0.50 0.47 0.03 2012-11-05 0.51 0.46 0.02 2012-11-07 0.51 0.41 0.00 Output would have only one row for each date.

Output: In the meantime, a time-window capability was added. See this link. In [1]: df = DataFrame({'B': range(5)}) In [2]: df.index = [Timestamp('20130101 09:00:00'), ...: Timestamp('20130101 09:00:02'), ...: Timestamp('20130101 09:00:03'), ...: Timestamp('20130101 09:00:05'), ...: Timestamp('20130101 09:00:06')] In [3]: df Out[3]: B 2013-01-01 09:00:00 0 2013-01-01 09:00:02 1 2013-01-01 09:00:03 2 2013-01-01 09:00:05 3 2013-01-01 09:00:06 4 In [4]: df.rolling(2, min_periods=1).sum() Out[4]: B 2013-01-01 09:00:00 0.0 2013-01-01 09:00:02 1.0 2013-01-01 09:00:03 3.0 2013-01-01 09:00:05 5.0 2013-01-01 09:00:06 7.0 In [5]: df.rolling('2s', min_periods=1).sum() Out[5]: B 2013-01-01 09:00:00 0.0 2013-01-01 09:00:02 1.0 2013-01-01 09:00:03 3.0 2013-01-01 09:00:05 3.0 2013-01-01 09:00:06 7.0


Instruction:
Pythonic way to convert a dictionary into namedtuple or another hashable dict-like?

I have a dictionary like: d = {'a': 1, 'b': 2, 'c': 3, 'd': 4} which I would like to convert to a namedtuple. My current approach is with the following code namedTupleConstructor = namedtuple('myNamedTuple', ' '.join(sorted(d.keys()))) nt= namedTupleConstructor(**d) which produces myNamedTuple(a=1, b=2, c=3, d=4) This works fine for me (I think), but am I missing a built-in such as... nt = namedtuple.from_dict() ? UPDATE: as discussed in the comments, my reason for wanting to convert my dictionary to a namedtuple is so that it becomes hashable, but still generally useable like a dict. UPDATE2: 4 years after I've posted this question, TLK posts a new answer recommending using the dataclass decorator that I think is really great. I think that's now what I would use going forward.

Output: To create the subclass, you may just pass the keys of a dict directly: from collections import namedtuple MyTuple = namedtuple('MyTuple', d) Now to create tuple instances from this dict, or any other dict with matching keys: my_tuple = MyTuple(**d) Beware: namedtuples compare on values only (ordered). They are designed to be a drop-in replacement for regular tuples, with named attribute access as an added feature. The field names will not be considered when making equality comparisons. It may not be what you wanted nor expected from the namedtuple type! This differs from dict equality comparisons, which do take into account the keys and also compare order agnostic. For readers who don't really need a type which is a subclass of tuple, there probably isn't much point to use a namedtuple in the first place. If you just want to use attribute access syntax on fields, it would be simpler and easier to create namespace objects instead: >>> from types import SimpleNamespace >>> SimpleNamespace(**d) namespace(a=1, b=2, c=3, d=4) my reason for wanting to convert my dictionary to a namedtuple is so that it becomes hashable, but still generally useable like a dict For a hashable "attrdict" like recipe, check out a frozen box: >>> from box import Box >>> b = Box(d, frozen_box=True) >>> hash(b) 7686694140185755210 >>> b.a 1 >>> b["a"] 1 >>> b["a"] = 2 BoxError: Box is frozen There may also be a frozen mapping type coming in a later version of Python, watch this draft PEP for acceptance or rejection: PEP 603 -- Adding a frozenmap type to collections


Instruction:
How do I list the files inside a python wheel?

I'm poking around the various options to setup.py for including non-python files, and they're somewhat less than intuitive. I'd like to be able to check the package generated by bdist_wheel to see what's actually in it--not so much to make sure that it will work (that's what tests are for) but to see the effects of the options I've set. How do I list the files contained in a .whl?

Output: You can take the wheel file change the extension to .zip and then extract the contents like any other zip file. from PEP 427 A wheel is a ZIP-format archive with a specially formatted file name and the .whl extension. Example the Django python package has a wheel file. Try Django-1.8.4-py2.py3-none-any.whl as an example. Their package contains non-python files if you wanted to see where they end up being stored in the archive. Code The following code works correctly using python2 and python3. It will list the files in any wheel package. I use the pep8 wheel package as an example, whose wheel can be downloaded with pip download --no-deps pep8==1.7.0. import pprint from zipfile import ZipFile path = 'pep8-1.7.0-py2.py3-none-any.whl' names = ZipFile(path).namelist() pprint.pprint(names) Output ['pep8.py', 'pep8-1.7.0.dist-info/DESCRIPTION.rst', 'pep8-1.7.0.dist-info/entry_points.txt', 'pep8-1.7.0.dist-info/metadata.json', 'pep8-1.7.0.dist-info/namespace_packages.txt', 'pep8-1.7.0.dist-info/top_level.txt', 'pep8-1.7.0.dist-info/WHEEL', 'pep8-1.7.0.dist-info/METADATA', 'pep8-1.7.0.dist-info/RECORD']


Instruction:
How can you find unused functions in Python code?

So you've got some legacy code lying around in a fairly hefty project. How can you find and delete dead functions? I've seen these two references: Find unused code and Tool to find unused functions in php project, but they seem specific to C# and PHP, respectively. Is there a Python tool that'll help you find functions that aren't referenced anywhere else in the source code (notwithstanding reflection/etc.)?

Output: In Python you can find unused code by using dynamic or static code analyzers. Two examples for dynamic analyzers are coverage and figleaf. They have the drawback that you have to run all possible branches of your code in order to find unused parts, but they also have the advantage that you get very reliable results. Alternatively, you can use static code analyzers that just look at your code, but don't actually run it. They run much faster, but due to Python's dynamic nature the results may contain false positives. Two tools in this category are pyflakes and vulture. Pyflakes finds unused imports and unused local variables. Vulture finds all kinds of unused and unreachable code. (Full disclosure: I'm the maintainer of Vulture.) The tools are available in the Python Package Index https://pypi.org/.


Instruction:
How to use youtube-dl from a python program?

I would like to access the result of the following shell command, youtube-dl -g "www.youtube.com/..." to print its output direct url to a file, from within a python program. This is what I have tried: import youtube-dl fromurl="www.youtube.com/..." geturl=youtube-dl.magiclyextracturlfromurl(fromurl) Is that possible? I tried to understand the mechanism in the source but got lost: youtube_dl/__init__.py, youtube_dl/youtube_DL.py, info_extractors ...

Output: It's not difficult and actually documented: import youtube_dl ydl = youtube_dl.YoutubeDL({'outtmpl': '%(id)s.%(ext)s'}) with ydl: result = ydl.extract_info( 'http://www.youtube.com/watch?v=BaW_jenozKc', download=False # We just want to extract the info ) if 'entries' in result: # Can be a playlist or a list of videos video = result['entries'][0] else: # Just a video video = result print(video) video_url = video['url'] print(video_url)


Instruction:
setup.py examples?

After studying this page: http://docs.python.org/distutils/builtdist.html I am hoping to find some setup.py files to study so as to make my own (with the goal of making a fedora rpm file). Could the s.o. community point me towards some good examples?

Output: Complete walkthrough of writing setup.py scripts here. (with some examples) If you'd like a real-world example, I could point you towards the setup.py scripts of a couple major projects. Django's is here, pyglet's is here. You can just browse the source of other projects for a file named setup.py for more examples. These aren't simple examples; the tutorial link I gave has those. These are more complex, but also more practical.


Instruction:
Indexing Pandas data frames: integer rows, named columns

Say df is a pandas dataframe. df.loc[] only accepts names df.iloc[] only accepts integers (actual placements) df.ix[] accepts both names and integers: When referencing rows, df.ix[row_idx, ] only wants to be given names. e.g. df = pd.DataFrame({'a' : ['one', 'two', 'three','four', 'five', 'six'], '1' : np.arange(6)}) df = df.ix[2:6] print(df) 1 a 2 2 three 3 3 four 4 4 five 5 5 six df.ix[0, 'a'] throws an error, it doesn't give return 'two'. When referencing columns, iloc is prefers integers, not names. e.g. df.ix[2, 1] returns 'three', not 2. (Although df.idx[2, '1'] does return 2). Oddly, I'd like the exact opposite functionality. Usually my column names are very meaningful, so in my code I reference them directly. But due to a lot of observation cleaning, the row names in my pandas data frames don't usually correspond to range(len(df)). I realize I can use: df.iloc[0].loc['a'] # returns three But it seems ugly! Does anyone know of a better way to do this, so that the code would look like this? df.foo[0, 'a'] # returns three In fact, is it possible to add on my own new method to pandas.core.frame.DataFrames, so e.g. df.idx(rows, cols) is in fact df.iloc[rows].loc[cols]?

Output: A very late answer but it amazed me that pandas still doesn't have such a function after all these years. If it irks you a lot, you can monkey-patch a custom indexer into the DataFrame: class XLocIndexer: def __init__(self, frame): self.frame = frame def __getitem__(self, key): row, col = key return self.frame.iloc[row][col] pd.core.indexing.IndexingMixin.xloc = property(lambda frame: XLocIndexer(frame)) # Usage df.xloc[0, 'a'] # one


Instruction:
Function parameter with colon

I just came across this function: def splitComma(line: str): splits = Utils.COMMA_DELIMITER.split(line) return "{}, {}".format(splits[1], splits[2]) I am aware that you can separate parameters by , or can set a value within a parameter like a=39 but I have not seen a colon like line:str. I have checked the function definition online but could not find anything like this. What does this colon mean?

Output: It's a function annotation; function arguments and the return value can be tagged with arbitrary Python expressions. Python itself ignores the annotation (other than saving it), but third-party tools can make use of them. In this case, it is intended as type hint: programs like mypy can analyze your code statically (that is, without running it, but only looking at the source code itself) to ensure that only str values are passed as arguments to splitComma. A fuller annotation to also specify the return type of the function: def splitComma(line: str) -> str: ... (Note that originally, function annotations weren't assumed to have any specific semantics. This is still true, but the overwhelming assumption these days is that the annotations provide type hints.)


Instruction:
Python - abs vs fabs

I noticed that in python there are two similar looking methods for finding the absolute value of a number: First abs(-5) Second import math math.fabs(-5) How do these methods differ?

Output: math.fabs() converts its argument to float if it can (if it can't, it throws an exception). It then takes the absolute value, and returns the result as a float. In addition to floats, abs() also works with integers and complex numbers. Its return type depends on the type of its argument. In [7]: type(abs(-2)) Out[7]: int In [8]: type(abs(-2.0)) Out[8]: float In [9]: type(abs(3+4j)) Out[9]: float In [10]: type(math.fabs(-2)) Out[10]: float In [11]: type(math.fabs(-2.0)) Out[11]: float In [12]: type(math.fabs(3+4j)) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/npe/<ipython-input-12-8368761369da> in <module>() ----> 1 type(math.fabs(3+4j)) TypeError: can't convert complex to float


Instruction:
What's with the integer cache maintained by the interpreter?

After dive into Python's source code, I find out that it maintains an array of PyInt_Objects ranging from int(-5) to int(256) (@src/Objects/intobject.c) A little experiment proves it: >>> a = 1 >>> b = 1 >>> a is b True >>> a = 257 >>> b = 257 >>> a is b False But if I run those code together in a py file (or join them with semi-colons), the result is different: >>> a = 257; b = 257; a is b True I'm curious why they are still the same object, so I digg deeper into the syntax tree and compiler, I came up with a calling hierarchy listed below: PyRun_FileExFlags() mod = PyParser_ASTFromFile() node *n = PyParser_ParseFileFlagsEx() //source to cst parsetoke() ps = PyParser_New() for (;;) PyTokenizer_Get() PyParser_AddToken(ps, ...) mod = PyAST_FromNode(n, ...) //cst to ast run_mod(mod, ...) co = PyAST_Compile(mod, ...) //ast to CFG PyFuture_FromAST() PySymtable_Build() co = compiler_mod() PyEval_EvalCode(co, ...) PyEval_EvalCodeEx() Then I added some debug code in PyInt_FromLong and before/after PyAST_FromNode, and executed a test.py: a = 257 b = 257 print "id(a) = %d, id(b) = %d" % (id(a), id(b)) the output looks like: DEBUG: before PyAST_FromNode name = a ival = 257, id = 176046536 name = b ival = 257, id = 176046752 name = a name = b DEBUG: after PyAST_FromNode run_mod PyAST_Compile ok id(a) = 176046536, id(b) = 176046536 Eval ok It means that during the cst to ast transform, two different PyInt_Objects are created (actually it's performed in the ast_for_atom() function), but they are later merged. I find it hard to comprehend the source in PyAST_Compile and PyEval_EvalCode, so I'm here to ask for help, I'll be appreciative if some one gives a hint?

Output: Python caches integers in the range [-5, 256], so integers in that range are usually but not always identical. What you see for 257 is the Python compiler optimizing identical literals when compiled in the same code object. When typing in the Python shell each line is a completely different statement, parsed and compiled separately, thus: >>> a = 257 >>> b = 257 >>> a is b False But if you put the same code into a file: $ echo 'a = 257 > b = 257 > print a is b' > testing.py $ python testing.py True This happens whenever the compiler has a chance to analyze the literals together, for example when defining a function in the interactive interpreter: >>> def test(): ... a = 257 ... b = 257 ... print a is b ... >>> dis.dis(test) 2 0 LOAD_CONST 1 (257) 3 STORE_FAST 0 (a) 3 6 LOAD_CONST 1 (257) 9 STORE_FAST 1 (b) 4 12 LOAD_FAST 0 (a) 15 LOAD_FAST 1 (b) 18 COMPARE_OP 8 (is) 21 PRINT_ITEM 22 PRINT_NEWLINE 23 LOAD_CONST 0 (None) 26 RETURN_VALUE >>> test() True >>> test.func_code.co_consts (None, 257) Note how the compiled code contains a single constant for the 257. In conclusion, the Python bytecode compiler is not able to perform massive optimizations (like statically typed languages), but it does more than you think. One of these things is to analyze usage of literals and avoid duplicating them. Note that this does not have to do with the cache, because it works also for floats, which do not have a cache: >>> a = 5.0 >>> b = 5.0 >>> a is b False >>> a = 5.0; b = 5.0 >>> a is b True For more complex literals, like tuples, it "doesn't work": >>> a = (1,2) >>> b = (1,2) >>> a is b False >>> a = (1,2); b = (1,2) >>> a is b False But the literals inside the tuple are shared: >>> a = (257, 258) >>> b = (257, 258) >>> a[0] is b[0] False >>> a[1] is b[1] False >>> a = (257, 258); b = (257, 258) >>> a[0] is b[0] True >>> a[1] is b[1] True (Note that constant folding and the peephole optimizer can change behaviour even between bugfix versions, so which examples return True or False is basically arbitrary and will change in the future). Regarding why you see that two PyInt_Object are created, I'd guess that this is done to avoid literal comparison. for example, the number 257 can be expressed by multiple literals: >>> 257 257 >>> 0x101 257 >>> 0b100000001 257 >>> 0o401 257 The parser has two choices: Convert the literals to some common base before creating the integer, and see if the literals are equivalent. then create a single integer object. Create the integer objects and see if they are equal. If yes, keep only a single value and assign it to all the literals, otherwise, you already have the integers to assign. Probably the Python parser uses the second approach, which avoids rewriting the conversion code and also it's easier to extend (for example it works with floats as well). Reading the Python/ast.c file, the function that parses all numbers is parsenumber, which calls PyOS_strtoul to obtain the integer value (for intgers) and eventually calls PyLong_FromString: x = (long) PyOS_strtoul((char *)s, (char **)&end, 0); if (x < 0 && errno == 0) { return PyLong_FromString((char *)s, (char **)0, 0); } As you can see here the parser does not check whether it already found an integer with the given value and so this explains why you see that two int objects are created, and this also means that my guess was correct: the parser first creates the constants and only afterward optimizes the bytecode to use the same object for equal constants. The code that does this check must be somewhere in Python/compile.c or Python/peephole.c, since these are the files that transform the AST into bytecode. In particular, the compiler_add_o function seems the one that does it. There is this comment in compiler_lambda: /* Make None the first constant, so the lambda can't have a docstring. */ if (compiler_add_o(c, c->u->u_consts, Py_None) < 0) return 0; So it seems like compiler_add_o is used to insert constants for functions/lambdas etc. The compiler_add_o function stores the constants into a dict object, and from this immediately follows that equal constants will fall in the same slot, resulting in a single constant in the final bytecode.


Instruction:
Module not found - "No module named"

Here's my Python folder structure -project ----src ------model --------order.py ------hello-world.py Under src, I have a folder named model, which has a Python file called order.py, whose contents follow: class SellOrder(object): def __init__(self,genericName,brandName): self.genericName = genericName self.brandName = brandName Next, my hello-world.py is inside the src folder, one level above order.py: import model.order.SellOrder order = SellOrder("Test","Test") print order.brandName Whenever I run python hello-world.py it results in the error Traceback (most recent call last): File "hello-world.py", line 1, in <module> import model.order.SellOrder ImportError: No module named model.order.SellOrder Is there anything I missed?

Output: All modules in Python have to have a certain directory structure. You can find details here. Create an empty file called __init__.py under the model directory, such that your directory structure would look something like that: . └── project └── src ├── hello-world.py └── model ├── __init__.py └── order.py Also in your hello-world.py file change the import statement to the following: from model.order import SellOrder That should fix it P.S.: If you are placing your model directory in some other location (not in the same directory branch), you will have to modify the python path using sys.path.


Instruction:
How accurate is python's time.sleep()?

I can give it floating point numbers, such as time.sleep(0.5) but how accurate is it? If i give it time.sleep(0.05) will it really sleep about 50 ms?

Output: The accuracy of the time.sleep function depends on your underlying OS's sleep accuracy. For non-real-time OSs like a stock Windows, the smallest interval you can sleep for is about 10-13ms. I have seen accurate sleeps within several milliseconds of that time when above the minimum 10-13ms. Update: Like mentioned in the docs cited below, it's common to do the sleep in a loop that will make sure to go back to sleep if it wakes you up early. I should also mention that if you are running Ubuntu you can try out a pseudo real-time kernel (with the RT_PREEMPT patch set) by installing the rt kernel package (at least in Ubuntu 10.04 LTS). Non-real-time Linux kernels have minimum sleep intervals much closer to 1ms than 10ms, but it varies in a non-deterministic manner.


Instruction:
Loading JSONL file as JSON objects

I want to load a JSONL file as JSON objects in python. Is there an easy way to do so?

Output: The splitlines would address that problem for you, so In general the code below will work for you: import json result = [json.loads(jline) for jline in jsonl_content.splitlines()] If that's the response object the result would be: result = [json.loads(jline) for jline in response.read().splitlines()] NOTE splitlines() treats line breaks differently than JSON's escape requirements. Specifically, splitlines() splits on Unicode Line Separator (\u2028), while JSON allows this character to remain unescaped in strings. When using json.dump with ensure_ascii=False, Python will emit \u2028 unescaped in strings.


Instruction:
Convert UTF-8 with BOM to UTF-8 with no BOM in Python

Two questions here. I have a set of files which are usually UTF-8 with BOM. I'd like to convert them (ideally in place) to UTF-8 with no BOM. It seems like codecs.StreamRecoder(stream, encode, decode, Reader, Writer, errors) would handle this. But I don't really see any good examples on usage. Would this be the best way to handle this? source files: Tue Jan 17$ file brh-m-157.json brh-m-157.json: UTF-8 Unicode (with BOM) text Also, it would be ideal if we could handle different input encoding wihtout explicitly knowing (seen ASCII and UTF-16). It seems like this should all be feasible. Is there a solution that can take any known Python encoding and output as UTF-8 without BOM? edit 1 proposed sol'n from below (thanks!) fp = open('brh-m-157.json','rw') s = fp.read() u = s.decode('utf-8-sig') s = u.encode('utf-8') print fp.encoding fp.write(s) This gives me the following error: IOError: [Errno 9] Bad file descriptor Newsflash I'm being told in comments that the mistake is I open the file with mode 'rw' instead of 'r+'/'r+b', so I should eventually re-edit my question and remove the solved part.

Output: This answer is for Python 2 Simply use the "utf-8-sig" codec: fp = open("file.txt") s = fp.read() u = s.decode("utf-8-sig") That gives you a unicode string without the BOM. You can then use s = u.encode("utf-8") to get a normal UTF-8 encoded string back in s. If your files are big, then you should avoid reading them all into memory. The BOM is simply three bytes at the beginning of the file, so you can use this code to strip them out of the file: import os, sys, codecs BUFSIZE = 4096 BOMLEN = len(codecs.BOM_UTF8) path = sys.argv[1] with open(path, "r+b") as fp: chunk = fp.read(BUFSIZE) if chunk.startswith(codecs.BOM_UTF8): i = 0 chunk = chunk[BOMLEN:] while chunk: fp.seek(i) fp.write(chunk) i += len(chunk) fp.seek(BOMLEN, os.SEEK_CUR) chunk = fp.read(BUFSIZE) fp.seek(-BOMLEN, os.SEEK_CUR) fp.truncate() It opens the file, reads a chunk, and writes it out to the file 3 bytes earlier than where it read it. The file is rewritten in-place. As easier solution is to write the shorter file to a new file like newtover's answer. That would be simpler, but use twice the disk space for a short period. As for guessing the encoding, then you can just loop through the encoding from most to least specific: def decode(s): for encoding in "utf-8-sig", "utf-16": try: return s.decode(encoding) except UnicodeDecodeError: continue return s.decode("latin-1") # will always work An UTF-16 encoded file wont decode as UTF-8, so we try with UTF-8 first. If that fails, then we try with UTF-16. Finally, we use Latin-1 — this will always work since all 256 bytes are legal values in Latin-1. You may want to return None instead in this case since it's really a fallback and your code might want to handle this more carefully (if it can).


Instruction:
How to copy directory recursively in python and overwrite all?

I'm trying to copy /home/myUser/dir1/ and all its contents (and their contents, etc.) to /home/myuser/dir2/ in python. Furthermore, I want the copy to overwrite everything in dir2/. It looks like distutils.dir_util.copy_tree might be the right tool for the job, but not sure if there's anything easier/more obvious to use for such a simple task. If it is the right tool, how do I use it? According to the docs there are 8 parameters that it takes. Do I have to pass all 8 are just src, dst and update, and if so, how (I'm brand new to Python). If there's something out there that's better, can someone give me an example and point me in the right direction?

Output: Notice: distutils has been deprecated and will be removed in Python 3.12. Consider looking for other answers at this question if you are looking for a post-3.12 solution. Original answer: You can use distutils.dir_util.copy_tree. It works just fine and you don't have to pass every argument, only src and dst are mandatory. However in your case you can't use a similar tool likeshutil.copytree because it behaves differently: as the destination directory must not exist this function can't be used for overwriting its contents. If you want to use the cp tool as suggested in the question comments beware that using the subprocess module is currently the recommended way for spawning new processes as you can see in the documentation of the os.system function.


Instruction:
python JSON only get keys in first level

I have a very long and complicated json object but I only want to get the items/keys in the first level! Example: { "1": "a", "3": "b", "8": { "12": "c", "25": "d" } } I want to get 1,3,8 as result! I found this code: for key, value in data.iteritems(): print key, value But it prints all keys (also 12 and 25)

Output: Just do a simple .keys() >>> dct = { ... "1": "a", ... "3": "b", ... "8": { ... "12": "c", ... "25": "d" ... } ... } >>> >>> dct.keys() ['1', '8', '3'] >>> for key in dct.keys(): print key ... 1 8 3 >>> If you need a sorted list: keylist = dct.keys() # this is of type `dict_key`, NOT a `list` keylist.sort() And if you want them as simple list, do this: list(dct_instance.keys())


Instruction:
Emulating Bash 'source' in Python

I have a script that looks something like this: export foo=/tmp/foo export bar=/tmp/bar Every time I build I run 'source init_env' (where init_env is the above script) to set up some variables. To accomplish the same in Python I had this code running, reg = re.compile('export (?P<name>\w+)(\=(?P<value>.+))*') for line in open(file): m = reg.match(line) if m: name = m.group('name') value = '' if m.group('value'): value = m.group('value') os.putenv(name, value) But then someone decided it would be nice to add a line like the following to the init_env file: export PATH="/foo/bar:/bar/foo:$PATH" Obviously my Python script fell apart. I could modify the Python script to handle this line, but then it'll just break later on when someone comes up with a new feature to use in the init_env file. The question is if there is an easy way to run a Bash command and let it modify my os.environ?

Output: The problem with your approach is that you are trying to interpret bash scripts. First you just try to interpret the export statement. But when people think they can use bash syntax they will. They will use variable expansion, conditionals, process substitutions. In the end you will have a full blown bash script interpreter with a gazillion bugs. Don't do that. Let Bash interpret the file for you and then collect the results. Here is a minimal example how to do so: #! /usr/bin/env python import os import pprint import shlex import subprocess command = shlex.split("bash -c 'source init_env && env'") proc = subprocess.Popen(command, stdout = subprocess.PIPE) for line in proc.stdout: (key, _, value) = line.partition("=") os.environ[key] = value proc.communicate() pprint.pprint(dict(os.environ)) Make sure that you handle errors. see here for how: "subprocess.Popen" - checking for success and errors Also read the documentation on subprocess. this will only capture variables set with the export statement, as env only prints exported variables. you can add set -a to treat all variables as exported. command = shlex.split("bash -c 'set -a && source init_env && env'") ^^^^^^ note that this code will not handle multi line variables. it will also not handle bash function definitions. perhaps better than calling bash source from inside python is to first let bash source the file and then run the python script #!/bin/bash source init_env /path/to/python_script.py here bash will source init_env with all the power and glory and quirks of bash. the python script will inherit the updated environment. note that again only exported variables will be inherited. you can force all variable assignments to be exported with set -a #!/bin/bash set -a source init_env /path/to/python_script.py another approach would be to tell the users that they can strictly only do key=value without any bash power. then use python configparser. this will have the advantage of simple init_env syntax and a rigorously tested config parser. but the disadvantage that the init_env will no longer be as expressive as bash config files can be.


Instruction:
Combine awaitables like Promise.all

In asynchronous JavaScript, it is easy to run tasks in parallel and wait for all of them to complete using Promise.all: async function bar(i) { console.log('started', i); await delay(1000); console.log('finished', i); } async function foo() { await Promise.all([bar(1), bar(2)]); } // This works too: async function my_all(promises) { for (let p of promises) await p; } async function foo() { await my_all([bar(1), bar(2), bar(3)]); } I tried to rewrite the latter in python: import asyncio async def bar(i): print('started', i) await asyncio.sleep(1) print('finished', i) async def aio_all(seq): for f in seq: await f async def main(): await aio_all([bar(i) for i in range(10)]) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close() But it executes my tasks sequentially. What is the simplest way to await multiple awaitables? Why doesn't my approach work?

Output: The equivalent would be using asyncio.gather: import asyncio async def bar(i): print('started', i) await asyncio.sleep(1) print('finished', i) async def main(): await asyncio.gather(*[bar(i) for i in range(10)]) loop = asyncio.get_event_loop() loop.run_until_complete(main()) loop.close() Why doesn't my approach work? Because when you await each item in seq, you block that coroutine. So in essence, you have synchronous code masquerading as async. If you really wanted to, you could implement your own version of asyncio.gather using loop.create_task or asyncio.ensure_future. EDIT The original answer used the lower-level asyncio.wait.


Instruction:
Why does my recursive function return None?

I have this function that calls itself: def get_input(): my_var = input('Enter "a" or "b": ') if my_var != "a" and my_var != "b": print('You didn\'t type "a" or "b". Try again.') get_input() else: return my_var print('got input:', get_input()) Now, if I input just "a" or "b", everything works fine: Type "a" or "b": a got input: a But, if I type something else and then "a" or "b", I get this: Type "a" or "b": purple You didn't type "a" or "b". Try again. Type "a" or "b": a got input: None I don't know why get_input() is returning None since it should only return my_var. Where is this None coming from and how do I fix my function?

Output: It is returning None because when you recursively call it: if my_var != "a" and my_var != "b": print('You didn\'t type "a" or "b". Try again.') get_input() ...you don't return the value. So while the recursion does happen, the return value gets discarded, and then you fall off the end of the function. Falling off the end of the function means that python implicitly returns None, just like this: >>> def f(x): ... pass >>> print(f(20)) None So, instead of just calling get_input() in your if statement, you need to return what the recursive call returns: if my_var != "a" and my_var != "b": print('You didn\'t type "a" or "b". Try again.') return get_input()


Instruction:
Python "from [dot]package import ..." syntax

Looking through a Django tutorial I saw the following syntax: from .models import Recipe, Ingredient, Instruction Can someone explain how the .models works / what it does exactly? Usually I have: from myapp.models import How does it work without the myapp part in front of .models?

Output: The . is a shortcut that tells it to search in the current package before the rest of the PYTHONPATH. So, if a same-named module Recipe exists somewhere else in your PYTHONPATH, it won't be loaded.


Instruction:
What does it mean for an object to be picklable (or pickle-able)?

Python docs mention this word a lot and I want to know what it means.

Output: It simply means it can be serialized by the pickle module. For a basic explanation of this, see What can be pickled and unpickled?. Pickling Class Instances provides more details, and shows how classes can customize the process.


Instruction:
How to parse multiple nested sub-commands using python argparse?

I am implementing a command line program which has interface like this: cmd [GLOBAL_OPTIONS] {command [COMMAND_OPTS]} [{command [COMMAND_OPTS]} ...] I have gone through the argparse documentation. I can implement GLOBAL_OPTIONS as optional argument using add_argument in argparse. And the {command [COMMAND_OPTS]} using Sub-commands. From the documentation it seems I can have only one sub-command. But as you can see I have to implement one or more sub-commands. What is the best way to parse such command line arguments useing argparse?

Output: @mgilson has a nice answer to this question. But problem with splitting sys.argv myself is that I lose the nice help message Argparse generates for the user. So I ended up doing this: import argparse ## This function takes the 'extra' attribute from global namespace and ## re-parses it to create separate namespaces for all other chained commands. def parse_extra (parser, namespace): namespaces = [] extra = namespace.extra while extra: n = parser.parse_args(extra) extra = n.extra namespaces.append(n) return namespaces argparser=argparse.ArgumentParser() subparsers = argparser.add_subparsers(help='sub-command help', dest='subparser_name') parser_a = subparsers.add_parser('command_a', help = "command_a help") ## Setup options for parser_a ## Add nargs="*" for zero or more other commands argparser.add_argument('extra', nargs = "*", help = 'Other commands') ## Do similar stuff for other sub-parsers Now after first parse all chained commands are stored in extra. I reparse it while it is not empty to get all the chained commands and create separate namespaces for them. And i get nicer usage string that argparse generates.


Instruction:
What exactly should be set in PYTHONPATH?

I'm going through and writing a setup doc for other developers at work for a python project and I've been reading up on the PYTHONPATH environment variable. I'm looking at my current development system and think I have a few things set wrong that is causing my IDE (IntelliJ) to behave incorrectly when looking up the python libraries. I've looked at the documentation here and here and I'm still unsure of what should actually be in the PYTHONPATH environment variable. I have PYTHONHOME pointed to C:\Python27. My current PYTHONPATH is set to PYTHONHOME. Should I also add the directories from sys.path? UPDATE: Based on the below information, PYTHONPATH does not need to be set unless there are non-standard libraries that you want python to be able to find by default. For instance, when I install wxPython from the installer it will add its libraries to PYTHONPATH. I do set PYTHONHOME to the root of the python installation so that I can add it to my system PATH environment variable so that I can run python from any where.

Output: You don't have to set either of them. PYTHONPATH can be set to point to additional directories with private libraries in them. PYTHONHOME sets the location of default libraries. Documentation: PYTHONHOME Change the location of the standard Python libraries. By default, the libraries are searched in prefix/lib/pythonversion and exec_prefix/lib/pythonversion, where prefix and exec_prefix are installation-dependent directories, both defaulting to /usr/local. When PYTHONHOME is set to a single directory, its value replaces both prefix and exec_prefix. To specify different values for these, set PYTHONHOME to prefix:exec_prefix. PYTHONPATH Augment the default search path for module files. The format is the same as the shell’s PATH: one or more directory pathnames separated by os.pathsep (e.g. colons on Unix or semicolons on Windows). Non-existent directories are silently ignored. In addition to normal directories, individual PYTHONPATH entries may refer to zipfiles containing pure Python modules (in either source or compiled form). Extension modules cannot be imported from zipfiles. The default search path is installation dependent, but generally begins with prefix/lib/pythonversion (see PYTHONHOME above). It is always appended to PYTHONPATH. An additional directory will be inserted in the search path in front of PYTHONPATH as described above under Interface options. The search path can be manipulated from within a Python program as the variable sys.path.


Instruction:
Quicksort with Python

I am totally new to python and I am trying to implement quicksort in it. Could someone please help me complete my code? I do not know how to concatenate the three arrays and print them. def sort(array=[12,4,5,6,7,3,1,15]): less = [] equal = [] greater = [] if len(array) > 1: pivot = array[0] for x in array: if x < pivot: less.append(x) if x == pivot: equal.append(x) if x > pivot: greater.append(x) sort(less) sort(pivot) sort(greater)

Output: def sort(array): """Sort the array by using quicksort.""" less = [] equal = [] greater = [] if len(array) > 1: pivot = array[0] for x in array: if x < pivot: less.append(x) elif x == pivot: equal.append(x) elif x > pivot: greater.append(x) # Don't forget to return something! return sort(less)+equal+sort(greater) # Just use the + operator to join lists # Note that you want equal ^^^^^ not pivot else: # You need to handle the part at the end of the recursion - when you only have one element in your array, just return the array. return array


Instruction:
inverting image in Python with OpenCV

I want to load a color image, convert it to grayscale, and then invert the data in the file. What I need: to iterate over the array in OpenCV and change every single value with this formula (it might be wrong but it seems reasonable for me): img[x,y] = abs(img[x,y] - 255) but I don't understand why doesn't it works: def inverte(imagem, name): imagem = abs(imagem - 255) cv2.imwrite(name, imagem) def inverte2(imagem, name): for x in np.nditer(imagem, op_flags=['readwrite']): x = abs(x - 255) cv2.imwrite(name, imagem) if __name__ == '__main__': nome = str(sys.argv[1]) image = cv2.imread(nome) gs_imagem = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) inverte(gs_imagem, "invertida.png") inverte2(gs_imagem, "invertida2.png") I don't want to do an explicit loop (I am trying to be more pythonic). I can see that in one image that got a white background it turned black, but only this it doesn't looks like the other colors are having much (if any) change.

Output: You almost did it. You were tricked by the fact that abs(imagem-255) will give a wrong result since your dtype is an unsigned integer. You have to do (255-imagem) in order to keep the integers unsigned: def inverte(imagem, name): imagem = (255-imagem) cv2.imwrite(name, imagem) As recommended in the answer by Eric Olmon, you can also invert the image using the bitwise_not function of OpenCV: imagem = cv2.bitwise_not(imagem)


Instruction:
How to Add Incremental Numbers to a New Column Using Pandas

I have this simplified dataframe: ID Fruit F1 Apple F2 Orange F3 Banana I want to add in the begining of the dataframe a new column df['New_ID'] which has the number 880 that increments by one in each row. The output should be simply like: New_ID ID Fruit 880 F1 Apple 881 F2 Orange 882 F3 Banana I tried the following: df['New_ID'] = ["880"] # but I want to do this without assigning it the list of numbers literally Any idea how to solve this? Thanks!

Output: Here: df = df.reset_index() df = df.rename(columns={"index":"New_ID"}) df['New_ID'] = df.index + 880


Instruction:
Check list monotonicity

How do I efficiently check list monotonicity? i.e. that it is either a non-decreasing or non-increasing set of ordered values? Examples: [0, 1, 2, 3, 3, 4] # This is a monotonically increasing list [4.3, 4.2, 4.2, -2] # This is a monotonically decreasing list [2, 3, 1] # This is neither

Output: Are repeated values (e.g. [1, 1, 2]) monotonic? If yes: def non_decreasing(L): return all(x<=y for x, y in zip(L, L[1:])) def non_increasing(L): return all(x>=y for x, y in zip(L, L[1:])) def monotonic(L): return non_decreasing(L) or non_increasing(L) If no: def strictly_increasing(L): return all(x<y for x, y in zip(L, L[1:])) def strictly_decreasing(L): return all(x>y for x, y in zip(L, L[1:])) def strictly_monotonic(L): return strictly_increasing(L) or strictly_decreasing(L)