Instruction:
Celery: When should you choose Redis as a message broker over RabbitMQ?

My rough understanding is that Redis is better if you need the in-memory key-value store feature, however I am not sure how that has anything to do with distributing tasks? Does that mean we should use Redis as a message broker IF we are already using it for something else?

Output: I've used both recently (2017-2018), and they are both super stable with Celery 4. So your choice can be based on the details of your hosting setup. If you must use Celery version 2 or version 3, go with RabbitMQ. Otherwise... If you are using Redis for any other reason, go with Redis If you are hosting at AWS, go with Redis so that you can use a managed Redis as service If you hate complicated installs, go with Redis If you already have RabbitMQ installed, stay with RabbitMQ In the past, I would have recommended RabbitMQ because it was more stable and easier to setup with Celery than Redis, but I don't believe that's true any more. Update 2019 AWS now has a managed service that is equivalent to RabbitMQ called Amazon MQ, which could reduce the headache of running this as a service in production. Please comment below if you have any experience with this and celery. Update 2025 The Celery Using Redis documentation lists some caveats for choosing Redis, which includes limitations with Visibility timeout - If a task isn’t acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed. Key eviction - In some circumstances a key could be removed unexpectedly. Group result ordering - For older versions of celery, results could be returned in an unexpected order. ... but if you're starting a new project it's unlikely that any situations will apply to you for some time, if ever.


Instruction:
Tkinter: How to use threads to preventing main event loop from "freezing"

I have a small GUI test with a "Start" button and a Progress bar. The desired behavior is: Click Start Progressbar oscillates for 5 seconds Progressbar stops The observed behavior is the "Start" button freezes for 5 seconds, then a Progressbar is displayed (no oscillation). Here is my code so far: class GUI: def __init__(self, master): self.master = master self.test_button = Button(self.master, command=self.tb_click) self.test_button.configure( text="Start", background="Grey", padx=50 ) self.test_button.pack(side=TOP) def progress(self): self.prog_bar = ttk.Progressbar( self.master, orient="horizontal", length=200, mode="indeterminate" ) self.prog_bar.pack(side=TOP) def tb_click(self): self.progress() self.prog_bar.start() # Simulate long running process t = threading.Thread(target=time.sleep, args=(5,)) t.start() t.join() self.prog_bar.stop() root = Tk() root.title("Test Button") main_ui = GUI(root) root.mainloop() Based on the information from Bryan Oakley here, I understand that I need to use threads. I tried creating a thread, but I'm guessing that since the thread is started from within the main thread, it doesn't help. I had the idea to place the logic portion in a different class, and instantiate the GUI from within that class, similar to the example code by A. Rodas here. My question: I can't figure out how to code it so that this command: self.test_button = Button(self.master, command=self.tb_click) calls a function that is located in the other class. Is this a Bad Thing to do or is it even possible? How would I create a 2nd class that can handle the self.tb_click? I tried following along to A. Rodas' example code which works beautifully. But I cannot figure out how to implement his solution in the case of a Button widget that triggers an action. If I should instead handle the thread from within the single GUI class, how would one create a thread that doesn't interfere with the main thread?

Output: When you join the new thread in the main thread, it will wait until the thread finishes, so the GUI will block even though you are using multithreading. If you want to place the logic portion in a different class, you can subclass Thread directly, and then start a new object of this class when you press the button. The constructor of this subclass of Thread can receive a Queue object and then you will be able to communicate it with the GUI part. So my suggestion is: Create a Queue object in the main thread Create a new thread with access to that queue Check periodically the queue in the main thread Then you have to solve the problem of what happens if the user clicks two times the same button (it will spawn a new thread with each click), but you can fix it by disabling the start button and enabling it again after you call self.prog_bar.stop(). import queue class GUI: # ... def tb_click(self): self.progress() self.prog_bar.start() self.queue = queue.Queue() ThreadedTask(self.queue).start() self.master.after(100, self.process_queue) def process_queue(self): try: msg = self.queue.get_nowait() # Show result of the task if needed self.prog_bar.stop() except queue.Empty: self.master.after(100, self.process_queue) class ThreadedTask(threading.Thread): def __init__(self, queue): super().__init__() self.queue = queue def run(self): time.sleep(5) # Simulate long running process self.queue.put("Task finished")


Instruction:
JSON serialize a dictionary with tuples as key

Is there a way in Python to serialize a dictionary that is using a tuple as key? e.g. a = {(1, 2): 'a'} simply using json.dumps(a) raises this error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.6/json/__init__.py", line 230, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.6/json/encoder.py", line 367, in encode chunks = list(self.iterencode(o)) File "/usr/lib/python2.6/json/encoder.py", line 309, in _iterencode for chunk in self._iterencode_dict(o, markers): File "/usr/lib/python2.6/json/encoder.py", line 268, in _iterencode_dict raise TypeError("key {0!r} is not a string".format(key)) TypeError: key (1, 2) is not a string

Output: You can't serialize that as json, json has a much less flexible idea about what counts as a dict key than python. You could transform the mapping into a sequence of key, value pairs, something like this: import json def remap_keys(mapping): return [{'key':k, 'value': v} for k, v in mapping.iteritems()] ... json.dumps(remap_keys({(1, 2): 'foo'})) >>> '[{"value": "foo", "key": [1, 2]}]'


Instruction:
Python executables: py2exe or PyInstaller?

To create executable files (windows) I assume that we should use one of them: Py2exe or PyInstaller. What are the difference between them?

Output: Py2exe and PyInstaller both are wrappers but here are few differences that I noticed, Py2exe is compatible with python2.4+ including python3.0 & 3.1 whereas PyInstaller is currently, compatible with python 2.7 and 3.3–3.5 As far I know, Py2exe didn't support signing whereas Pyinstaller has support for signing from version 1.4 In PyInstaller it is easy to create one exe, By default both create a bunch of exes & dlls. In py2exe its easier to embed manifest file in exe, useful for run as administrator mode in windows vista and beyond. Pyinstaller is modular and has a feature of hooks to include files in the build that you like. I don't know about this feature in py2exe. Hope this helps you in your decision making. [Update] - It looks like PyInstaller is actively developed (https://github.com/pyinstaller/pyinstaller/) and released. py2exe is still using sourceforge and its release cycle is very random on pypi there is no build after 2014 and their code show development in 2017 as well (https://sourceforge.net/p/py2exe/svn/HEAD/tree/trunk/py2exe-3/py2exe/). So, I recommend using pyinstaller till the time py2exe stabilizes its release cycle in favor of developers.


Instruction:
patching a class yields "AttributeError: Mock object has no attribute" when accessing instance attributes

The Problem Using mock.patch with autospec=True to patch a class is not preserving attributes of instances of that class. The Details I am trying to test a class Bar that instantiates an instance of class Foo as a Bar object attribute called foo. The Bar method under test is called bar; it calls method foo of the Foo instance belonging to Bar. In testing this, I am mocking Foo, as I only want to test that Bar is accessing the correct Foo member: import unittest from mock import patch class Foo(object): def __init__(self): self.foo = 'foo' class Bar(object): def __init__(self): self.foo = Foo() def bar(self): return self.foo.foo class TestBar(unittest.TestCase): @patch('foo.Foo', autospec=True) def test_patched(self, mock_Foo): Bar().bar() def test_unpatched(self): assert Bar().bar() == 'foo' The classes and methods work just fine (test_unpatched passes), but when I try to Foo in a test case (tested using both nosetests and pytest) using autospec=True, I encounter "AttributeError: Mock object has no attribute 'foo'" 19:39 $ nosetests -sv foo.py test_patched (foo.TestBar) ... ERROR test_unpatched (foo.TestBar) ... ok ====================================================================== ERROR: test_patched (foo.TestBar) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/mock.py", line 1201, in patched return func(*args, **keywargs) File "/home/vagrant/dev/constellation/test/foo.py", line 19, in test_patched Bar().bar() File "/home/vagrant/dev/constellation/test/foo.py", line 14, in bar return self.foo.foo File "/usr/local/lib/python2.7/dist-packages/mock.py", line 658, in __getattr__ raise AttributeError("Mock object has no attribute %r" % name) AttributeError: Mock object has no attribute 'foo' Indeed, when I print out mock_Foo.return_value.__dict__, I can see that foo is not in the list of children or methods: {'_mock_call_args': None, '_mock_call_args_list': [], '_mock_call_count': 0, '_mock_called': False, '_mock_children': {}, '_mock_delegate': None, '_mock_methods': ['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__'], '_mock_mock_calls': [], '_mock_name': '()', '_mock_new_name': '()', '_mock_new_parent': <MagicMock name='Foo' spec='Foo' id='38485392'>, '_mock_parent': <MagicMock name='Foo' spec='Foo' id='38485392'>, '_mock_wraps': None, '_spec_class': <class 'foo.Foo'>, '_spec_set': None, 'method_calls': []} My understanding of autospec is that, if True, the patch specs should apply recursively. Since foo is indeed an attribute of Foo instances, should it not be patched? If not, how do I get the Foo mock to preserve the attributes of Foo instances? NOTE: This is a trivial example that shows the basic problem. In reality, I am mocking a third party module.Class -- consul.Consul -- whose client I instantiate in a Consul wrapper class that I have. As I don't maintain the consul module, I can't modify the source to suit my tests (I wouldn't really want to do that anyway). For what it's worth, consul.Consul() returns a consul client, which has an attribute kv -- an instance of consul.Consul.KV. kv has a method get, which I am wrapping in an instance method get_key in my Consul class. After patching consul.Consul, the call to get fails because of AttributeError: Mock object has no attribute kv. Resources Already Checked: http://mock.readthedocs.org/en/latest/helpers.html#autospeccing http://mock.readthedocs.org/en/latest/patch.html

Output: No, autospeccing cannot mock out attributes set in the __init__ method of the original class (or in any other method). It can only mock out static attributes, everything that can be found on the class. Otherwise, the mock would have to create an instance of the class you tried to replace with a mock in the first place, which is not a good idea (think classes that create a lot of real resources when instantiated). The recursive nature of an auto-specced mock is then limited to those static attributes; if foo is a class attribute, accessing Foo().foo will return an auto-specced mock for that attribute. If you have a class Spam whose eggs attribute is an object of type Ham, then the mock of Spam.eggs will be an auto-specced mock of the Ham class. The documentation you read explicitly covers this: A more serious problem is that it is common for instance attributes to be created in the __init__ method and not to exist on the class at all. autospec can’t know about any dynamically created attributes and restricts the api to visible attributes. You should just set the missing attributes yourself: @patch('foo.Foo', autospec=Foo) def test_patched(self, mock_Foo): mock_Foo.return_value.foo = 'foo' Bar().bar() or create a subclass of your Foo class for testing purposes that adds the attribute as a class attribute: class TestFoo(foo.Foo): foo = 'foo' # class attribute @patch('foo.Foo', autospec=TestFoo) def test_patched(self, mock_Foo): Bar().bar()


Instruction:
drop trailing zeros from decimal

I have a long list of Decimals and that I have to adjust by factors of 10, 100, 1000,..... 1000000 depending on certain conditions. When I multiply them there is sometimes a useless trailing zero (though not always) that I want to get rid of. For example... from decimal import Decimal # outputs 25.0, PROBLEM! I would like it to output 25 print Decimal('2.5') * 10 # outputs 2567.8000, PROBLEM! I would like it to output 2567.8 print Decimal('2.5678') * 1000 Is there a function that tells the decimal object to drop these insignificant zeros? The only way I can think of doing this is to convert to a string and replace them using regular expressions. Should probably mention that I am using python 2.6.5 EDIT senderle's fine answer made me realize that I occasionally get a number like 250.0 which when normalized produces 2.5E+2. I guess in these cases I could try to sort them out and convert to a int

Output: There's probably a better way of doing this, but you could use .rstrip('0').rstrip('.') to achieve the result that you want. Using your numbers as an example: >>> s = str(Decimal('2.5') * 10) >>> print s.rstrip('0').rstrip('.') if '.' in s else s 25 >>> s = str(Decimal('2.5678') * 1000) >>> print s.rstrip('0').rstrip('.') if '.' in s else s 2567.8 And here's the fix for the problem that @gerrit pointed out in the comments: >>> s = str(Decimal('1500')) >>> print s.rstrip('0').rstrip('.') if '.' in s else s 1500


Instruction:
Flask and uWSGI - unable to load app 0 (mountpoint='') (callable not found or import error)

I get the below error when I try and start Flask using uWSGI. Here is how I start: > # cd .. > root@localhost:# uwsgi --socket 127.0.0.1:6000 --file /path/to/folder/run.py --callable app - -processes 2 Here is my directory structure: -/path/to/folder/run.py -|app -|__init__.py -|views.py -|templates -|static Contents of /path/to/folder/run.py if __name__ == '__main__': from app import app #app.run(debug = True) app.run() Contents of /path/to/folder/app/__init__.py import os from flask import Flask from flask.ext.sqlalchemy import SQLAlchemy from flask.ext.login import LoginManager #from flaskext.babel import Babel from config import basedir app = Flask(__name__) app.config.from_object('config') #app.config.from_pyfile('babel.cfg') db = SQLAlchemy(app) login_manager = LoginManager() login_manager.setup_app(app) login_manager.login_view = 'login' login_manager.login_message = u"Please log in to access this page." from app import views *** Operational MODE: preforking *** unable to find "application" callable in file /path/to/folder/run.py unable to load app 0 (mountpoint='') (callable not found or import error) *** no app loaded. going in full dynamic mode *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI worker 1 (pid: 26972, cores: 1) spawned uWSGI worker 2 (pid: 26973, cores: 1)

Output: uWSGI doesn't load your app as __main__, so it never will find the app (since that only gets loaded when the app is run as name __main__). Thus, you need to import it outside of the if __name__ == "__main__": block. Really simple change: from app import app as application # for example, should be app if __name__ == "__main__": application.run() Now you can run the app directly with python run.py or run it through uWSGI the way you have it. NOTE: if you set --callable myapp, you'd need to change it from as application to myapp (by default uwsgi expects application


Instruction:
How to pass an entire list as command line argument in Python?

I was trying to pass two lists containing integers as arguments to a python code. But sys.argv[i] gets the parameters as a list of string. Input would look like, $ python filename.py [2,3,4,5] [1,2,3,4] I found the following hack to convert the list. strA = sys.argv[1].replace('[', ' ').replace(']', ' ').replace(',', ' ').split() strB = sys.argv[2].replace('[', ' ').replace(']', ' ').replace(',', ' ').split() A = [float(i) for i in strA] B = [float (i) for i in strB] Is there a better way to do this?

Output: Command line arguments are always passed as strings. You will need to parse them into your required data type yourself. >>> input = "[2,3,4,5]" >>> map(float, input.strip('[]').split(',')) [2.0, 3.0, 4.0, 5.0] >>> A = map(float, input.strip('[]').split(',')) >>> print(A, type(A)) ([2.0, 3.0, 4.0, 5.0], <type 'list'>) There are libraries like argparse and click that let you define your own argument type conversion but argparse treats "[2,3,4]" the same as [ 2 , 3 , 4 ] so I doubt it will be useful. edit Jan 2019 This answer seems to get a bit of action still so I'll add another option taken directly from the argparse docs. You can use action=append to allow repeated arguments to be collected into a single list. >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action='append') >>> parser.parse_args('--foo 1 --foo 2'.split()) Namespace(foo=['1', '2']) In this case you would pass --foo ? once for each list item. Using OPs example: python filename.py --foo 2 --foo 3 --foo 4 --foo 5 would result in foo=[2,3,4,5]


Instruction:
Take screenshot of full page with Selenium Python with chromedriver

After trying out various approaches... I have stumbled upon this page to take full-page screenshot with chromedriver, selenium and python. The original code is here. (and I copy the code in this posting below) It uses PIL and it works great! However, there is one issue... which is it captures fixed headers and repeats for the whole page and also misses some parts of the page during page change. sample url to take a screenshot: http://www.w3schools.com/js/default.asp How to avoid the repeated headers with this code... Or is there any better option which uses python only... ( i don't know java and do not want to use java). Please see the screenshot of the current result and sample code below. test.py """ This script uses a simplified version of the one here: https://snipt.net/restrada/python-selenium-workaround-for-full-page-screenshot-using-chromedriver-2x/ It contains the *crucial* correction added in the comments by Jason Coutu. """ import sys from selenium import webdriver import unittest import util class Test(unittest.TestCase): """ Demonstration: Get Chrome to generate fullscreen screenshot """ def setUp(self): self.driver = webdriver.Chrome() def tearDown(self): self.driver.quit() def test_fullpage_screenshot(self): ''' Generate document-height screenshot ''' #url = "http://effbot.org/imagingbook/introduction.htm" url = "http://www.w3schools.com/js/default.asp" self.driver.get(url) util.fullpage_screenshot(self.driver, "test.png") if __name__ == "__main__": unittest.main(argv=[sys.argv[0]]) util.py import os import time from PIL import Image def fullpage_screenshot(driver, file): print("Starting chrome full page screenshot workaround ...") total_width = driver.execute_script("return document.body.offsetWidth") total_height = driver.execute_script("return document.body.parentNode.scrollHeight") viewport_width = driver.execute_script("return document.body.clientWidth") viewport_height = driver.execute_script("return window.innerHeight") print("Total: ({0}, {1}), Viewport: ({2},{3})".format(total_width, total_height,viewport_width,viewport_height)) rectangles = [] i = 0 while i < total_height: ii = 0 top_height = i + viewport_height if top_height > total_height: top_height = total_height while ii < total_width: top_width = ii + viewport_width if top_width > total_width: top_width = total_width print("Appending rectangle ({0},{1},{2},{3})".format(ii, i, top_width, top_height)) rectangles.append((ii, i, top_width,top_height)) ii = ii + viewport_width i = i + viewport_height stitched_image = Image.new('RGB', (total_width, total_height)) previous = None part = 0 for rectangle in rectangles: if not previous is None: driver.execute_script("window.scrollTo({0}, {1})".format(rectangle[0], rectangle[1])) print("Scrolled To ({0},{1})".format(rectangle[0], rectangle[1])) time.sleep(0.2) file_name = "part_{0}.png".format(part) print("Capturing {0} ...".format(file_name)) driver.get_screenshot_as_file(file_name) screenshot = Image.open(file_name) if rectangle[1] + viewport_height > total_height: offset = (rectangle[0], total_height - viewport_height) else: offset = (rectangle[0], rectangle[1]) print("Adding to stitched image with offset ({0}, {1})".format(offset[0],offset[1])) stitched_image.paste(screenshot, offset) del screenshot os.remove(file_name) part = part + 1 previous = rectangle stitched_image.save(file) print("Finishing chrome full page screenshot workaround...") return True

Output: How it works: set browser height as longest as you can... #coding=utf-8 import time from selenium import webdriver from selenium.webdriver.chrome.options import Options def test_fullpage_screenshot(self): # please note that we MUST use headless mode chrome_options = Options() chrome_options.add_argument('--headless') chrome_options.add_argument('--start-maximized') driver = webdriver.Chrome(chrome_options=chrome_options) driver.get("yoururlxxx") time.sleep(2) height = driver.execute_script('return document.documentElement.scrollHeight') width = driver.execute_script('return document.documentElement.scrollWidth') driver.set_window_size(width, height) # the trick time.sleep(2) driver.save_screenshot("screenshot1.png") driver.quit() if __name__ == "__main__": test_fullpage_screenshot()


Instruction:
Split a string into N equal parts?

I have a string I would like to split into N equal parts. For example, imagine I had a string with length 128 and I want to split it in to 4 chunks of length 32 each; i.e., first 32 chars, then the second 32 and so on. How can I do this?

Output: import textwrap print(textwrap.wrap("123456789", 2)) #prints ['12', '34', '56', '78', '9'] Note: be careful with whitespace etc - this may or may not be what you want. """Wrap a single paragraph of text, returning a list of wrapped lines. Reformat the single paragraph in 'text' so it fits in lines of no more than 'width' columns, and return a list of wrapped lines. By default, tabs in 'text' are expanded with string.expandtabs(), and all other whitespace characters (including newline) are converted to space. See TextWrapper class for available keyword args to customize wrapping behaviour. """


Instruction:
How to get every first element in 2 dimensional list

I have a list like this: a = [[4.0, 4, 4.0], [3.0, 3, 3.6], [3.5, 6, 4.8]] I want an outcome like this (EVERY first element in the list): 4.0, 3.0, 3.5 I tried a[::1][0], but it doesn't work

Output: You can get the index [0] from each element in a list comprehension >>> [i[0] for i in a] [4.0, 3.0, 3.5]


Instruction:
pip3: command not found

I want to install TensorFlow following these instructions: https://web.archive.org/web/20170627102751/https://www.tensorflow.org/versions/r0.12/get_started/os_setup#pip_installation But when I try this code on terminal, it returns an error. $ sudo pip3 install --upgrade $TF_BINARY_URL sudo: pip3: command not found So I installed Homebrew and tried to uninstall and reinstall python3-pip, but didn't work. MakotonoMacBook-ea:~ makotomiyazaki$ brew uninstall python3-pip Error: No such keg: /usr/local/Cellar/python3-pip MakotonoMacBook-ea:~ makotomiyazaki$ brew install python3-pip Error: No available formula with the name "python3-pip" ==> Searching for a previously deleted formula... Warning: homebrew/core is shallow clone. To get complete history run: git -C "$(brew --repo homebrew/core)" fetch --unshallow What should I do for getting pip3? My OS is macOS High Sierra, and I have Python 3.6.2 already installed. EDIT: I tried python3 -m pip and what's returned was this: The directory '/Users/makotomiyazaki/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. The directory '/Users/makotomiyazaki/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. You must give at least one requirement to install (see "pip help install") I also tried which pip3, but just I don't know if it worked... MakotonoMacBook-ea:~ makotomiyazaki$ sudo which pip3 install --upgrade $TF_BINARY_URL /usr/bin/install

Output: You would need to install pip3. On Linux, run first sudo apt update. Then the command would be: sudo apt install python3-pip On Mac, using brew, first brew install python3 Then brew postinstall python3 Try calling pip3 -V to see if it worked.


Instruction:
Pandas column access w/column names containing spaces

If I import or create a pandas column that contains no spaces, I can access it as such: from pandas import DataFrame df1 = DataFrame({'key': ['b', 'b', 'a', 'c', 'a', 'a', 'b'], 'data1': range(7)}) df1.data1 which would return that series for me. If, however, that column has a space in its name, it isn't accessible via that method: from pandas import DataFrame df2 = DataFrame({'key': ['a','b','d'], 'data 2': range(3)}) df2.data 2 # <--- not the droid I'm looking for. I know I can access it using .xs(): df2.xs('data 2', axis=1) There's got to be another way. I've googled it like mad and can't think of any other way to google it. I've read all 96 entries here on SO that contain "column" and "string" and "pandas" and could find no previous answer. Is this the only way, or is there something better?

Output: I think the default way is to use the bracket method instead of the dot notation. import pandas as pd df1 = pd.DataFrame({ 'key': ['b', 'b', 'a', 'c', 'a', 'a', 'b'], 'dat a1': range(7) }) df1['dat a1'] The other methods, like exposing it as an attribute are more for convenience.


Instruction:
Using Python's ftplib to get a directory listing, portably

You can use ftplib for full FTP support in Python. However the preferred way of getting a directory listing is: # File: ftplib-example-1.py import ftplib ftp = ftplib.FTP("www.python.org") ftp.login("anonymous", "ftplib-example-1") data = [] ftp.dir(data.append) ftp.quit() for line in data: print "-", line Which yields: $ python ftplib-example-1.py - total 34 - drwxrwxr-x 11 root 4127 512 Sep 14 14:18 . - drwxrwxr-x 11 root 4127 512 Sep 14 14:18 .. - drwxrwxr-x 2 root 4127 512 Sep 13 15:18 RCS - lrwxrwxrwx 1 root bin 11 Jun 29 14:34 README -> welcome.msg - drwxr-xr-x 3 root wheel 512 May 19 1998 bin - drwxr-sr-x 3 root 1400 512 Jun 9 1997 dev - drwxrwxr-- 2 root 4127 512 Feb 8 1998 dup - drwxr-xr-x 3 root wheel 512 May 19 1998 etc ... I guess the idea is to parse the results to get the directory listing. However this listing is directly dependent on the FTP server's way of formatting the list. It would be very messy to write code for this having to anticipate all the different ways FTP servers might format this list. Is there a portable way to get an array filled with the directory listing? (The array should only have the folder names.)

Output: Try using ftp.nlst(dir). However, note that if the folder is empty, it might throw an error: files = [] try: files = ftp.nlst() except ftplib.error_perm as resp: if str(resp) == "550 No files found": print "No files in this directory" else: raise for f in files: print f


Instruction:
Why don't my south migrations work?

First, I create my database. create database mydb; I add "south" to installed Apps. Then, I go to this tutorial: http://south.aeracode.org/docs/tutorial/part1.html The tutorial tells me to do this: $ py manage.py schemamigration wall --initial >>> Created 0001_initial.py. You can now apply this migration with: ./manage.py migrate wall Great, now I migrate. $ py manage.py migrate wall But it gives me this error... django.db.utils.DatabaseError: (1146, "Table 'fable.south_migrationhistory' doesn't exist") So I use Google (which never works. hence my 870 questions asked on Stackoverflow), and I get this page: http://groups.google.com/group/south-users/browse_thread/thread/d4c83f821dd2ca1c Alright, so I follow that instructions >> Drop database mydb; >> Create database mydb; $ rm -rf ./wall/migrations $ py manage.py syncdb But when I run syncdb, Django creates a bunch of tables. Yes, it creates the south_migrationhistory table, but it also creates my app's tables. Synced: > django.contrib.admin > django.contrib.auth > django.contrib.contenttypes > django.contrib.sessions > django.contrib.sites > django.contrib.messages > south > fable.notification > pagination > timezones > fable.wall > mediasync > staticfiles > debug_toolbar Not synced (use migrations): - (use ./manage.py migrate to migrate these) Cool....now it tells me to migrate these. So, I do this: $ py manage.py migrate wall The app 'wall' does not appear to use migrations. Alright, so fine. I'll add wall to initial migrations. $ py manage.py schemamigration wall --initial Then I migrate: $ py manage.py migrate wall You know what? It gives me this BS: _mysql_exceptions.OperationalError: (1050, "Table 'wall_content' already exists") Sorry, this is really pissing me off. Can someone help ? thanks. How do I get South to work and sync correctly with everything? The only thing I can think of is remove my app from INSTALLED_APPS, then run syncdb, then add it back on. That is SO SILLY.

Output: South allows you to create migrations when you first start out with a new app and the tables haven't been added to the database yet, as well as creating migrations for legacy apps that already have tables in the database. The key is to know when to do what. Your first mistake was when you deleted your migrations, as soon as you did that, and then ran syncdb, Django didn't know that you wanted south to manage that app anymore, so it created the tables for you. When you created your initial migrations and then ran migrate, south was trying to create tables that django already created, and thus your error. At this point you have two options. Delete the tables for the wall app from your database and then run $ py manage.py migrate wall This will run the migration and create your tables. Fake out the initial migration run $ py manage.py migrate wall 0001 --fake This will tell south that you already have the tables on the database so just fake it, which will add a row to the south_migrationhistory table, so that the next time you run a migrate it will know that the first migration has already been run. Setting up a brand new project and no database create your database add south to installed apps run syncdb, this will add the django and south tables to the database add your apps for each app run python manage.py schemamigration app_name --initial this will create the initial migration files for your app then run south migrate python manage.py migrate app_name this will add the tables to the database. Setting up a legacy project and database add south to installed apps run syncdb, this will add the south tables to the database for each of your apps run python manage.py schemamigration app_name --initial This will create your initial migrations for each of your apps run python manage.py migrate app_name 0001 --fake , this will fake out south, it won't do anything to the database for those models, it will just add records to the south_migrationhistory table so that the next time you want to create a migration, you are all set. Setting up a legacy project and no database create database add south to installed apps for each of your apps run python manage.py schemamigration app_name --initial This will create your initial migrations run syncdb, this will add any apps that don't have migrations to the database. then run south migrate python manage.py migrate this will run all migrations for your apps. Now that you are setup with south, you can start using south to manage model changes to those apps. The most common command to run is python manage.py schemamigration app_name migration_name --auto that will look at the last migration you ran and it will find the changes and build out a migration file for you. Then you just need to run python manage.py migrate and it alter your database for you.


Instruction:
How to pipe input to python line by line from linux program?

I want to pipe the output of ps -ef to python line by line. The script I am using is this (first.py) - #! /usr/bin/python import sys for line in sys.argv: print line Unfortunately, the "line" is split into words separated by whitespace. So, for example, if I do echo "days go by and still" | xargs first.py the output I get is ./first.py days go by and still How to write the script such that the output is ./first.py days go by and still ?

Output: Instead of using command line arguments I suggest reading from standard input (stdin). Python has a simple idiom for iterating over lines at stdin: import sys for line in sys.stdin: sys.stdout.write(line) My usage example (with above's code saved to iterate-stdin.py): $ echo -e "first line\nsecond line" | python iterate-stdin.py first line second line With your example: $ echo "days go by and still" | python iterate-stdin.py days go by and still


Instruction:
Tell if Python is in interactive mode

In a Python script, is there any way to tell if the interpreter is running in interactive mode? This would be useful so that, for instance, when you run an interactive Python session and import a module, different code can be executed. What I mean is something like this: if __name__ == "__main__": # do stuff elif __pythonIsInteractive__: # do other stuff else: exit()

Output: Python <= 3.12 __main__.__file__ doesn't exist in the interactive interpreter for Python versions <= 3.12: import __main__ print(hasattr(__main__, "__file__")) This also goes for code run via python -c, but not python -m.


Instruction:
How do you base-64 encode a PNG image for use in a data-uri in a CSS file?

I want to base-64 encode a PNG file, to include it in a data:url in my stylesheet. How can I do that? I’m on a Mac, so something on the Unix command line would work great. A Python-based solution would also be grand.

Output: This should do it in Python: import base64 binary_fc = open(filepath, 'rb').read() # fc aka file_content base64_utf8_str = base64.b64encode(binary_fc).decode('utf-8') ext = filepath.split('.')[-1] dataurl = f'data:image/{ext};base64,{base64_utf8_str}' Thanks to @cnst comment, we need the prefix data:image/{ext};base64, Thanks to @ramazanpolat answer, we need the decode('utf-8')


Instruction:
Check if a process is running or not on Windows?

I am trying to create a python script which I will later run as a service. Now I want to run a particular part of the code only when iTunes is running. I understand from some research that polling the entire command list and then searching for the application for that list is expensive. I found out that processes on UNIX-based operating systems create a lock file to notify that a program is currently running, at which point we can use os.stat(location_of_file) to check if the file exists to determine if a program is running or not. Is there a similar lock file created on Windows? If not what are the various ways in Python by which we can determine if a process is running or not? I am using python 2.7 and iTunes COM interface.

Output: I found psutil very slow. A much more performant solution: import win32gui window_titles = set() def winEnumHandler( hwnd, ctx ): if win32gui.IsWindowVisible( hwnd ): window_titles.add(win32gui.GetWindowText( hwnd ) ) win32gui.EnumWindows( winEnumHandler, None ) print(window_titles)


Instruction:
How to extract all the emojis from text?

Consider the following list: a_list = ['🤔 🙈 me así, bla es se 😌 ds 💕👭👙'] How can I extract in a new list all the emojis inside a_list?: new_lis = ['🤔 🙈 😌 💕 👭 👙'] I tried to use regex, but I do not have all the possible emojis encodings.

Output: You can use the emoji library. You can check if a single codepoint is an emoji codepoint by checking if it is contained in emoji.UNICODE_EMOJI. import emoji def extract_emojis(s): return ''.join(c for c in s if c in emoji.UNICODE_EMOJI['en'])


Instruction:
How to setup pip to download from mirror repository by default?

I am forced to download python packages from local mirror PyPi repository. I do this by using the -i and --trusted-host options. Whole installation command looks like this: pip install -i https://sampleurl.com/pypi-remote/simple --trusted-host sample.host.com package_name Having to type in that options each time is kinda annoying though (in reality those are long URL's). I've tried to create get_package.bat file (I'm working on Windows 10) with following content: pip install -i https://sampleurl.com/pypi-remote/simple --trusted-host sample.host.com "%1" It works perfectly fine, although when I wanted to execute pip search command, it turned out to be useless since it has hard-coded install command and there is no way to use it with search. Is there any way in which I can setup pip to download from mirror repository by default, so that I can execute pip install package_name or pip search package_name without any additional options? Eventually I could try making .bat file that would take 2 parameters like this: pip %1 -i https://sampleurl.com/pypi-remote/simple --trusted-host sample.host.com "%2" But I wonder if there's more "elegant" way to do this.

Output: using pip config, on user or global level. I have /etc/pip.conf configured like this: [global] index=https://my-company/nexus/repository/pypi-group/pypi index-url=https://my-company/nexus/repository/pypi-group/simple trusted-host=my-company but you can configure this using pip config on user or global level, something like: pip config --user set global.index https://my-company/nexus/repository/pypi-group/pypi pip config --user set global.index-url https://my-company/nexus/repository/pypi-group/simple pip config --user set global.trusted-host my-company #NOTES --index-url is used by pip install --index is used by pip search


Instruction:
set matplotlib 3d plot aspect ratio

import matplotlib.pyplot as plt from mpl_toolkits.mplot3d import Axes3D Setting the aspect ratio works for 2d plots: ax = plt.axes() ax.plot([0,1], [0,10]) ax.set_aspect('equal', 'box') But it does not work for 3d: ax = plt.axes(projection='3d') ax.plot([0,1], [0,1], [0,10]) ax.set_aspect('equal', 'box') How do I set the aspect ratio for 3d?

Output: My understanding is basically that this isn't implemented yet (see this bug in GitHub). I'm also hoping that it is implemented soon. See This link for a possible solution (I haven't tested it myself).


Instruction:
How to run multiple functions at the same time?

I'm trying to run 2 functions at the same time. def func1(): print('Working') def func2(): print('Working') func1() func2() Does anyone know how to do this?

Output: Do this: from threading import Thread def func1(): print('Working') def func2(): print("Working") if __name__ == '__main__': Thread(target = func1).start() Thread(target = func2).start()


Instruction:
Create a two-dimensional array with two one-dimensional arrays

My function (name CovexHull(point)) accepts the argument as a two-dimensional array: hull = ConvexHull(points) Session In [1]: points.ndim Out[1]: 2 In [2]: points.shape Out[2]: (10, 2) In [3]: points Out[3]: array([[ 0. , 0. ], [ 1. , 0.8], [ 0.9, 0.8], [ 0.9, 0.7], [ 0.9, 0.6], [ 0.8, 0.5], [ 0.8, 0.5], [ 0.7, 0.5], [ 0.1, 0. ], [ 0. , 0. ]]) points is a NumPy array with ndim 2. I have two different NumPy arrays (tp and fp) like below: In [4]: fp.ndim Out[4]: 1 In [5]: fp.shape Out[5]: (10,) In [6]: fp Out[6]: array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.4, 0.5, 0.6, 0.9, 1. ]) How can I create a two-dimensional NumPy array effectively (like points mentioned above) with tp and fp?

Output: If you wish to combine two 10 element one-dimensional arrays into a two-dimensional array, np.vstack((tp, fp)).T will do it. np.vstack((tp, fp)) will return an array of shape (2, 10), and the T attribute returns the transposed array with shape (10, 2) (i.e., with the two one-dimensional arrays forming columns rather than rows). >>> tp = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> tp.ndim 1 >>> tp.shape (10,) >>> fp = np.array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) >>> fp.ndim 1 >>> fp.shape (10,) >>> combined = np.vstack((tp, fp)).T >>> combined array([[ 0, 10], [ 1, 11], [ 2, 12], [ 3, 13], [ 4, 14], [ 5, 15], [ 6, 16], [ 7, 17], [ 8, 18], [ 9, 19]]) >>> combined.ndim 2 >>> combined.shape (10, 2)


Instruction:
Iterate over all combinations of values in multiple lists in Python

Given multiple list of possibly varying length, I want to iterate over all combinations of values, one item from each list. For example: first = [1, 5, 8] second = [0.5, 4] Then I want the output of to be: combined = [(1, 0.5), (1, 4), (5, 0.5), (5, 4), (8, 0.5), (8, 4)] I want to iterate over the combined list. How do I get this done?

Output: itertools.product should do the trick. >>> import itertools >>> list(itertools.product([1, 5, 8], [0.5, 4])) [(1, 0.5), (1, 4), (5, 0.5), (5, 4), (8, 0.5), (8, 4)] Note that itertools.product returns an iterator, so you don't need to convert it into a list if you are only going to iterate over it once. eg. for x in itertools.product([1, 5, 8], [0.5, 4]): # do stuff


Instruction:
Running windows shell commands with python

How can we interact with OS shell using Python ? I want to run windows cmd commands via python. How can it be achieved ?

Output: The newer subprocess.check_output and similar commands are supposed to replace os.system. See this page for details. While I can't test this on Windows (because I don't have access to any Windows machines), the following should work: from subprocess import check_output check_output("dir C:", shell=True) check_output returns a string of the output from your command. Alternatively, subprocess.call just runs the command and returns the status of the command (usually 0 if everything is okay). Also note that, in python 3, that string output is now bytes output. If you want to change this into a string, you need something like from subprocess import check_output check_output("dir C:", shell=True).decode() If necessary, you can tell it the kind of encoding your program outputs. The default is utf-8, which typically works fine, but other standard options are here. Also note that @bluescorpion says in the comments that Windows 10 needs a trailing backslash, as in check_output("dir C:\\", shell=True). The double backslash is needed because \ is a special character in python, so it has to be escaped. (Also note that even prefixing the string with r doesn't help if \ is the very last character of the string — r"dir C:\" is a syntax error, though r"dir C:\ " is not.)


Instruction:
How would I compute exactly 30 days into the past with Python (down to the minute)?

In Python, I'm attempting to retrieve the date/time that is exactly 30 days (30*24hrs) into the past. At present, I'm simply doing: >>> import datetime >>> start_date = datetime.date.today() + datetime.timedelta(-30) Which returns a datetime object, but with no time data: >>> start_date.year 2009 >>> start_date.hour Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'datetime.date' object has no attribute 'hour'

Output: You want to use a datetime object instead of just a date object: start_date = datetime.datetime.now() - datetime.timedelta(30) date just stores a date and time just a time. datetime is a date with a time.


Instruction:
How to save a figure remotely with pylab?

I'm trying to generate a figure at a remote computer with the command pylab.savefig. But I got such error: Unable to access the X Display, is $DISPLAY set properly? How can I save the figure properly?

Output: By default, matplotlib will use something like the TkAgg backend. This requires an X-server to be running. While you can just use X-forwarding, there will be a noticeable lag as matplotlib tries to connect with the remote X-server. If you don't need to interact with the plot, it's often nicer to speed things up by avoiding an X-connection entirely. If you want to make a plot without needing an X-server at all, use the Agg backend instead. E.g. do something like this: import matplotlib matplotlib.use('Agg') # Must be before importing matplotlib.pyplot or pylab! import matplotlib.pyplot as plt fig = plt.figure() plt.plot(range(10)) fig.savefig('temp.png') If you want this to be the default behavior, you can modify your matplotlibrc file to use the Agg backend by default. See this article for more information.


Instruction:
How to I display why some tests where skipped while using py.test?

I am using skipIf() from unittest for skipping tests in certain conditions. @unittest.skipIf(condition), "this is why I skipped them!") How do I tell py.test to display skipping conditions? I know that for unittest I need to enable the verbose mode (-v) but the same parameter added to py.test increase the verbosity by still does not display the skip reasons.

Output: When you run py.test, you can pass -rsx to report skipped tests. From py.test --help: -r chars show extra test summary info as specified by chars (f)ailed, (E)error, (s)skipped, (x)failed, (X)passed. Also see this part of the documentation about skipping: http://doc.pytest.org/en/latest/skipping.html


Instruction:
Class with too many parameters: better design strategy?

I am working with models of neurons. One class I am designing is a cell class which is a topological description of a neuron (several compartments connected together). It has many parameters but they are all relevant, for example: Number of axon segments, apical bifibrications, somatic length, somatic diameter, apical length, branching randomness, branching length and so on and so on... there are about 15 parameters in total! I can set all these to some default value but my class looks crazy with several lines for parameters. This kind of thing must happen occasionally to other people too, is there some obvious better way to design this or am I doing the right thing? As you can see this class has a huge number of parameters (>15) but they are all used and are necessary to define the topology of a cell. The problem essentially is that the physical object they create is very complex. How would experienced programmers do this differently to avoid so many parameters in the definition? class LayerV(__Cell): def __init__(self,somatic_dendrites=10,oblique_dendrites=10, somatic_bifibs=3,apical_bifibs=10,oblique_bifibs=3, L_sigma=0.0,apical_branch_prob=1.0, somatic_branch_prob=1.0,oblique_branch_prob=1.0, soma_L=30,soma_d=25,axon_segs=5,myelin_L=100, apical_sec1_L=200,oblique_sec1_L=40,somadend_sec1_L=60, ldecf=0.98): import random import math #make main the regions: axon=Axon(n_axon_seg=axon_segs) soma=Soma(diam=soma_d,length=soma_L) main_apical_dendrite=DendriticTree(bifibs= apical_bifibs,first_sec_L=apical_sec1_L, L_sigma=L_sigma,L_decrease_factor=ldecf, first_sec_d=9,branch_prob=apical_branch_prob) #make the somatic denrites somatic_dends=self.dendrite_list(num_dends=somatic_dendrites, bifibs=somatic_bifibs,first_sec_L=somadend_sec1_L, first_sec_d=1.5,L_sigma=L_sigma, branch_prob=somatic_branch_prob,L_decrease_factor=ldecf) #make oblique dendrites: oblique_dends=self.dendrite_list(num_dends=oblique_dendrites, bifibs=oblique_bifibs,first_sec_L=oblique_sec1_L, first_sec_d=1.5,L_sigma=L_sigma, branch_prob=oblique_branch_prob,L_decrease_factor=ldecf) #connect axon to soma: axon_section=axon.get_connecting_section() self.soma_body=soma.body soma.connect(axon_section,region_end=1) #connect apical dendrite to soma: apical_dendrite_firstsec=main_apical_dendrite.get_connecting_section() soma.connect(apical_dendrite_firstsec,region_end=0) #connect oblique dendrites to apical first section: for dendrite in oblique_dends: apical_location=math.exp(-5*random.random()) #for now connecting randomly but need to do this on some linspace apsec=dendrite.get_connecting_section() apsec.connect(apical_dendrite_firstsec,apical_location,0) #connect dendrites to soma: for dend in somatic_dends: dendsec=dend.get_connecting_section() soma.connect(dendsec,region_end=random.random()) #for now connecting randomly but need to do this on some linspace #assign public sections self.axon_iseg=axon.iseg self.axon_hill=axon.hill self.axon_nodes=axon.nodes self.axon_myelin=axon.myelin self.axon_sections=[axon.hill]+[axon.iseg]+axon.nodes+axon.myelin self.soma_sections=[soma.body] self.apical_dendrites=main_apical_dendrite.all_sections+self.seclist(oblique_dends) self.somatic_dendrites=self.seclist(somatic_dends) self.dendrites=self.apical_dendrites+self.somatic_dendrites self.all_sections=self.axon_sections+[self.soma_sections]+self.dendrites

Output: This approach may be suited in your specific case, but it definitely has its downsides, see is kwargs an antipattern? Try this approach: class Neuron(object): def __init__(self, **kwargs): prop_defaults = { "num_axon_segments": 0, "apical_bifibrications": "fancy default", ... } for (prop, default) in prop_defaults.iteritems(): setattr(self, prop, kwargs.get(prop, default)) You can then create a Neuron like this: n = Neuron(apical_bifibrications="special value")


Instruction:
How to do an else (default) in match-case?

Python recently has released match-case in version 3.10. The question is how can we do a default case in Python? I can do if/elif but don't know how to do else. Below is the code: x = "hello" match x: case "hi": print(x) case "hey": print(x) default: print("not matched") I added this default myself. I want to know the method to do this in Python.

Output: You can define a default case in Python. For this you use a wild card (_). The following code demonstrates it: x = "hello" match x: case "hi": print(x) case "hey": print(x) case _: print("not matched")


Instruction:
Is there a possibility to execute a Python script while being in interactive mode

Normally you can execute a Python script for example: python myscript.py, but if you are in the interactive mode, how is it possible to execute a Python script on the filesystem? >>> exec(File) ??? It should be possible to execute the script more than one time.

Output: Use execfile('script.py') but it only work on python 2.x, if you are using 3.0 try exec(open('script.py').read())


Instruction:
What is the fastest way to parse large XML docs in Python?

I am currently running the following code based on Chapter 12.5 of the Python Cookbook: from xml.parsers import expat class Element(object): def __init__(self, name, attributes): self.name = name self.attributes = attributes self.cdata = '' self.children = [] def addChild(self, element): self.children.append(element) def getAttribute(self,key): return self.attributes.get(key) def getData(self): return self.cdata def getElements(self, name=''): if name: return [c for c in self.children if c.name == name] else: return list(self.children) class Xml2Obj(object): def __init__(self): self.root = None self.nodeStack = [] def StartElement(self, name, attributes): element = Element(name.encode(), attributes) if self.nodeStack: parent = self.nodeStack[-1] parent.addChild(element) else: self.root = element self.nodeStack.append(element) def EndElement(self, name): self.nodeStack.pop() def CharacterData(self,data): if data.strip(): data = data.encode() element = self.nodeStack[-1] element.cdata += data def Parse(self, filename): Parser = expat.ParserCreate() Parser.StartElementHandler = self.StartElement Parser.EndElementHandler = self.EndElement Parser.CharacterDataHandler = self.CharacterData ParserStatus = Parser.Parse(open(filename).read(),1) return self.root I am working with XML documents of about 1 GB in size. Does anyone know a faster way to parse these?

Output: I looks to me as if you do not need any DOM capabilities from your program. I would second the use of the (c)ElementTree library. If you use the iterparse function of the cElementTree module, you can work your way through the xml and deal with the events as they occur. Note however, Fredriks advice on using cElementTree iterparse function: to parse large files, you can get rid of elements as soon as you’ve processed them: for event, elem in iterparse(source): if elem.tag == "record": ... process record elements ... elem.clear() The above pattern has one drawback; it does not clear the root element, so you will end up with a single element with lots of empty child elements. If your files are huge, rather than just large, this might be a problem. To work around this, you need to get your hands on the root element. The easiest way to do this is to enable start events, and save a reference to the first element in a variable: # get an iterable context = iterparse(source, events=("start", "end")) # turn it into an iterator context = iter(context) # get the root element event, root = context.next() for event, elem in context: if event == "end" and elem.tag == "record": ... process record elements ... root.clear() The lxml.iterparse() does not allow this. The previous does not work on Python 3.7, consider the following way to get the first element. import xml.etree.ElementTree as ET # Get an iterable. context = ET.iterparse(source, events=("start", "end")) for index, (event, elem) in enumerate(context): # Get the root element. if index == 0: root = elem if event == "end" and elem.tag == "record": # ... process record elements ... root.clear()


Instruction:
Why does ActivePython exist?

What's ActivePython actually about? From what I've read it's just standard Python with OpenSSL and PyWin32 (on Windows). No big deal I guess; I could install them in matter of minutes, and most people don't need them anyway. All other mentioned libraries (zlib, bzip2, SQLite 3, Tkinter, ElementTree, ctypes, and multiprocessing) are part of the core Python distribution. Next up, the tag-line "ActivePython is the industry-standard Python distribution", isn't core Python distribution "industry-standard" (whatever that means?)? And the weirdest thing, is that ActiveState bundles it with crappy PythonWin, and not their own most-awesome Python editor/IDE, Komodo. What gives? I actually never got to installing ActivePython, so maybe I don't know something, but it seems pretty irrelevant, and I see the name quite often on forums or here.

Output: It's a packaging, or "distribution", of Python, with some extras -- not (anywhere) quite as "Sumo" as Enthought's huge distribution of "Python plus everything", but still in a similar vein (and it first appeared much earlier). I don't think you're missing anything in particular, except perhaps the fact that David Ascher (Python enthusiast and my coauthor in the Python Cookbook) used to be CTO at ActiveState (and so no doubt internally pushed Python to go with other dynamic languages ActiveState focuses on), but he's gone now (he's CEO at the Mozilla-owned firm that deals with email and similar forms of communication -- Thunderbird and the like, in terms of programs). No doubt some firms prefer to purchase a distribution with commercially available support contracts, like ActivePython, just because that's the way some purchasing departments in several enterprises (and/or their IT departments) are used to work. Unless you care about such issues, I don't think you're missing anything by giving ActiveState's Python distribution a pass;-). (I feel similarly about costly Enterprise distributions of Linux, vs. Debian or Ubuntu or the like -- but then I'm not in purchasing, nor in an IT department, nor do I work for a very traditional enterprise anyway;-) )


Instruction:
Why is code using intermediate variables faster than code without?

I have encountered this weird behavior and failed to explain it. These are the benchmarks: py -3 -m timeit "tuple(range(2000)) == tuple(range(2000))" 10000 loops, best of 3: 97.7 usec per loop py -3 -m timeit "a = tuple(range(2000)); b = tuple(range(2000)); a==b" 10000 loops, best of 3: 70.7 usec per loop How come comparison with variable assignment is faster than using a one liner with temporary variables by more than 27%? By the Python docs, garbage collection is disabled during timeit so it can't be that. Is it some sort of an optimization? The results may also be reproduced in Python 2.x though to lesser extent. Running Windows 7/10, CPython 3.5.1 all the way to 3.10.1, Intel i7 3.40 GHz, 64 bit both OS and Python. Seems like a different machine I've tried running at Intel i7 3.60 GHz with Python 3.5.0 does not reproduce the results. Running using the same Python process with timeit.timeit() @ 10000 loops produced 0.703 and 0.804 respectively. Still shows although to lesser extent. (~12.5%)

Output: My results were similar to yours: the code that uses intermediate variables was consistently at least 10-20 % faster in Python 3.4. However when I used IPython on the very same Python 3.4 interpreter, I got these results: In [1]: %timeit -n10000 -r20 tuple(range(2000)) == tuple(range(2000)) 10000 loops, best of 20: 74.2 µs per loop In [2]: %timeit -n10000 -r20 a = tuple(range(2000)); b = tuple(range(2000)); a==b 10000 loops, best of 20: 75.7 µs per loop Notably, I never managed to get even close to the 74.2 µs for the former when I used -mtimeit from the command line. So this Heisenbug turned out to be something quite interesting. I decided to run the command with strace and indeed there is something fishy going on: % strace -o withoutvars python3 -m timeit "tuple(range(2000)) == tuple(range(2000))" 10000 loops, best of 3: 134 usec per loop % strace -o withvars python3 -mtimeit "a = tuple(range(2000)); b = tuple(range(2000)); a==b" 10000 loops, best of 3: 75.8 usec per loop % grep mmap withvars|wc -l 46 % grep mmap withoutvars|wc -l 41149 Now that is a good reason for the difference. The code that does not use variables causes the mmap system call be called almost 1000x more than the one that uses intermediate variables. The withoutvars is full of mmap/munmap for a 256k region; these same lines are repeated over and over again: mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f32e56de000 munmap(0x7f32e56de000, 262144) = 0 mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f32e56de000 munmap(0x7f32e56de000, 262144) = 0 mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f32e56de000 munmap(0x7f32e56de000, 262144) = 0 The mmap call seems to be coming from the function _PyObject_ArenaMmap from Objects/obmalloc.c; the obmalloc.c also contains the macro ARENA_SIZE, which is #defined to be (256 << 10) (that is 262144); similarly the munmap matches the _PyObject_ArenaMunmap from obmalloc.c. obmalloc.c says that Prior to Python 2.5, arenas were never free()'ed. Starting with Python 2.5, we do try to free() arenas, and use some mild heuristic strategies to increase the likelihood that arenas eventually can be freed. Thus these heuristics and the fact that Python object allocator releases these free arenas as soon as they're emptied lead to python3 -mtimeit 'tuple(range(2000)) == tuple(range(2000))' triggering pathological behaviour where one 256 kiB memory area is re-allocated and released repeatedly; and this allocation happens with mmap/munmap, which is comparatively costly as they're system calls - furthermore, mmap with MAP_ANONYMOUS requires that the newly mapped pages must be zeroed - even though Python wouldn't care. The behaviour is not present in the code that uses intermediate variables, because it is using slightly more memory and no memory arena can be freed as some objects are still allocated in it. That is because timeit will make it into a loop not unlike for n in range(10000) a = tuple(range(2000)) b = tuple(range(2000)) a == b Now the behaviour is that both a and b will stay bound until they're *reassigned, so in the second iteration, tuple(range(2000)) will allocate a 3rd tuple, and the assignment a = tuple(...) will decrease the reference count of the old tuple, causing it to be released, and increase the reference count of the new tuple; then the same happens to b. Therefore after the first iteration there are always at least 2 of these tuples, if not 3, so the thrashing doesn't occur. Most notably it cannot be guaranteed that the code using intermediate variables is always faster - indeed in some setups it might be that using intermediate variables will result in extra mmap calls, whereas the code that compares return values directly might be fine. Someone asked that why this happens, when timeit disables garbage collection. It is indeed true that timeit does it: Note By default, timeit() temporarily turns off garbage collection during the timing. The advantage of this approach is that it makes independent timings more comparable. This disadvantage is that GC may be an important component of the performance of the function being measured. If so, GC can be re-enabled as the first statement in the setup string. For example: However, the garbage collector of Python is only there to reclaim cyclic garbage, i.e. collections of objects whose references form cycles. It is not the case here; instead these objects are freed immediately when the reference count drops to zero.


Instruction:
Asking "is hashable" about a Python value

I am interested in taking an arbitrary dict and copying it into a new dict, mutating it along the way. One mutation I would like to do is swap keys and value. Unfortunately, some values are dicts in their own right. However, this generates a "unhashable type: 'dict'" error. I don't really mind just stringifying the value and giving it the key. But, I'd like to be able to do something like this: for key in olddict: if hashable(olddict[key]): newdict[olddict[key]] = key else newdict[str(olddict[key])] = key Is there a clean way to do this that doesn't involve trapping an exception and parsing the message string for "unhashable type" ?

Output: Python 3.x Use collections.abc.Hashable or typing.Hashable. >>> import typing >>> isinstance({}, typing.Hashable) False >>> isinstance(0, typing.Hashable) True Note: both are the same one, the latter is simply an alias of the former. Also note that collections.Hashable was removed in Python 3.10+ (deprecated since 3.7). Python 2.6+ (an original answer) Since Python 2.6 you can use the abstract base class collections.Hashable: >>> import collections >>> isinstance({}, collections.Hashable) False >>> isinstance(0, collections.Hashable) True This approach is also mentioned briefly in the documentation for __hash__. Doing so means that not only will instances of the class raise an appropriate TypeError when a program attempts to retrieve their hash value, but they will also be correctly identified as unhashable when checking isinstance(obj, collections.Hashable) (unlike classes which define their own __hash__() to explicitly raise TypeError).


Instruction:
What is the use of stub files (.pyi ) in python?

I am trying to understand the lower level implementations of python 3. There is one module named _posixsubprocess used by the subprocess module. I tried to find the location of this module in my system and found that it's a stub file. Could someone guide me as I have no idea about what are the stub files and how are they implemented at the lower level?

Output: _posixsubprocess The file you are referencing is a Python module written in C. It's not a "stub" file. The real implementation can be found in the stdlib at Modules/_posixsubprocess.c. You can see how writing a C/C++ extension is written by having a look at Building C and C++ Extensions. This should help you understanding the code in _posixsubprocess.c. In order to add type-hints to that file (which is an "Extension Module" as it is written in C), the type hints are added to a "stub" file with the extension .pyi. That file can be found in the typeshed which is a collection of stub files. The typeshed also contains stubs for third-party modules which is a historical remnant. That is no longer needed since PEP-561 has been adopted. Concerning stub/pyi files Stub files contain type-hinting information of normal Python modules. The full official documentation can be found in the section about stub-files in PEP-484. For example, if you have a Python module mymodule.py like this: def myfunction(name): return "Hello " + name Then you can add type-hints via a stub-file mymodule.pyi. Note that here the ellipsis (...) is part of the syntax, so the code-block below really shows the complete file contents: def myfunction(name: str) -> str: ... They look very similar to C header files in that they contain only the function signatures, but their use is purely optional. You can also add type hints directly in the .py module like the following: def myfunction(name: str) -> str: return "Hello " + name But there are some cases where you want to keep them separate in stubs: You want to keep your code Python 2 compatible and don't like the # type: ... comment syntax You use function annotations for something else but still want to use type-hints You are adding type-hints into an existing code-base and want to keep code-churn in existing files minimal


Instruction:
Evaluating pytorch models: `with torch.no_grad` vs `model.eval()`

When I want to evaluate the performance of my model on the validation set, is it preferred to use with torch.no_grad: or model.eval()?

Output: TL;DR: Use both. They do different things, and have different scopes. with torch.no_grad - disables tracking of gradients in autograd. model.eval() changes the forward() behaviour of the module it is called upon eg, it disables dropout and has batch norm use the entire population statistics with torch.no_grad The torch.autograd.no_grad documentation says: Context-manager that disabled [sic] gradient calculation. Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True. In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True. model.eval() The nn.Module.eval documentation says: Sets the module in evaluation mode. This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc. The creator of pytorch said the documentation should be updated to suggest the usage of both, and I raised the pull request.


Instruction:
Python: if not val, vs if val is None

I've always coded in the style of if not value, however, a few guides have brought to my attention that while this style works, it seems to have 2 potential problems: It's not completely readable; if value is None is surely more understandable. This can have implications later (and cause subtle bugs), since things like [] and 0 will evaluate to False as well. I am also starting to apply this idea to other comparisons, such as: if not value vs if value is False if not value vs if value is [] And so goes the list... The question is, how far do you go with the principle? Where to draw the line, while keeping your code safe? Should I always use the if value is None style no matter what?

Output: Use a comparison to None if that's what you want. Use if not value if you just want to check if the value is considered False (empty list, None, False). I find if not value to be cleaner looking and Pythonic. Also, be careful with lists. You should not use is when comparing for an empty list. If you know you're getting a list, use if <list> to check if it has any contents (or len(<list>)). Try typing this into the interpreter: >>> a = [] >>> a is [] False This is because the temporary list you just made has a different address in memory than the one stored at 'a'. You don't see this with None, False, or True because these are all values that are singletons (they all refer to the same section of memory) so using the is keyword works. You'll also find that CPython interns strings so the following works. >>> 'a' is 'a' True You should not rely on this. It is an implementation detail and this is not specified to work with every version of Python.


Instruction:
Python, what does an underscore before parenthesis do

Looking through some of the Django code in authentication forms I noticed the following syntax: label=_("Username") Normally I would have just used a pair of quotes around the string. Can someone explain what the underscore and parenthesis around "Username" do?

Output: The _ is the name of a callable (function, callable object). It's usually used for the gettext function, for example in Django: from django.utils.translation import gettext as _ print _("Hello!") # Will print Hello! if the current language is English # "Bonjour !" in French # ¡Holà! in Spanish, etc. As the doc says: Python’s standard library gettext module installs _() into the global namespace, as an alias for gettext(). In Django, we have chosen not to follow this practice, for a couple of reasons: [...] The underscore character (_) is used to represent “the previous result” in Python’s interactive shell and doctest tests. Installing a global _() function causes interference. Explicitly importing gettext() as _() avoids this problem. Even if it's a convention, it may not be the case in your code. But be reassured, 99.9% of the time _ is an alias for gettext :)


Instruction:
Attaching a decorator to all functions within a class

Is there a way to bind a decorator to all functions within a class generically, rather than explicitly stating it for every function? I suppose it then becomes a kind of aspect, rather than a decorator and it does feel a bit odd, but was thinking for something like timing or auth it'd be pretty neat.

Output: The cleanest way to do this, or to do other modifications to a class definition, is to define a metaclass. Alternatively, just apply your decorator at the end of the class definition using inspect: import inspect class Something: def foo(self): pass for name, fn in inspect.getmembers(Something, inspect.isfunction): setattr(Something, name, decorator(fn)) In practice of course you'll want to apply your decorator more selectively. As soon as you want to decorate all but one method you'll discover that it is easier and more flexible just to use the decorator syntax in the traditional way.


Instruction:
What is pipenv [dev-packages] section for?

I'm learning pipenv and there's something I don't quite understand. Apparently your Pipfile can contain two sections: [packages] ... [dev-packages] .... As I understand it, the packages section is where the packages that you install are listed. But what is the dev-packages section for? How is it different from the packages section?

Output: This section is for development requirements. So stuff like linters, unit test libraries, etc. All that is not needed on the user's machine. To install a package as dev-requirement add -d to install command (i.e. pipenv install -d ...), to install the dev-requirements section add -d to sync command (i.e. pipenv sync -d ...).


Instruction:
Is it bad practice to include non-validating methods in a pydantic model?

I'm using pydantic 1.3 to validate models for an API I am writing. Is it common/good practice to include arbitrary methods in a class that inherits from pydantic.BaseModel? I need some helper methods associated with the objects and I am trying to decide whether I need a "handler" class. These models are being converted to JSON and sent to a restful service that I am also writing. My model looks like this: class Foo(pydantic.BaseModel): name: str bar: int baz: int Is it poor practice to do something like: class Foo(pydantic.BaseModel): name: str bar: int baz: int def add_one(self): self.bar += 1 It makes some sense to me, but I can't find an example of anyone doing this.

Output: Yes, it's fine. We should probably document it. The only problem comes when you have a field name which conflicts with the method, but that's not a problem if you know what your data looks like. Also, it's possible to over object orient your code, but you're a long way from that.


Instruction:
array.array versus numpy.array

If you are creating a 1d array in Python, is there any benefit to using the NumPy package?

Output: It all depends on what you plan to do with the array. If all you're doing is creating arrays of simple data types and doing I/O, the array module will do just fine. If, on the other hand, you want to do any kind of numerical calculations, the array module doesn't provide any help with that. NumPy (and SciPy) give you a wide variety of operations between arrays and special functions that are useful not only for scientific work but for things like advanced image manipulation or in general anything where you need to perform efficient calculations with large amounts of data. Numpy is also much more flexible, e.g. it supports arrays of any type of Python objects, and is also able to interact "natively" with your own objects if they conform to the array interface.


Instruction:
How to implement custom indentation when pretty-printing with the JSON module?

So I'm using Python 2.7, using the json module to encode the following data structure: 'layer1': { 'layer2': { 'layer3_1': [ long_list_of_stuff ], 'layer3_2': 'string' } } My problem is that I'm printing everything out using pretty printing, as follows: json.dumps(data_structure, indent=2) Which is great, except I want to indent it all, except for the content in "layer3_1" — It's a massive dictionary listing coordinates, and as such, having a single value set on each one makes pretty printing create a file with thousands of lines, with an example as follows: { "layer1": { "layer2": { "layer3_1": [ { "x": 1, "y": 7 }, { "x": 0, "y": 4 }, { "x": 5, "y": 3 }, { "x": 6, "y": 9 } ], "layer3_2": "string" } } } What I really want is something similar to the following: { "layer1": { "layer2": { "layer3_1": [{"x":1,"y":7},{"x":0,"y":4},{"x":5,"y":3},{"x":6,"y":9}], "layer3_2": "string" } } } I hear it's possible to extend the json module: Is it possible to set it to only turn off indenting when inside the "layer3_1" object? If so, would somebody please tell me how?

Output: (Note: The code in this answer only works with json.dumps() which returns a JSON formatted string, but not with json.dump() which writes directly to file-like objects. There's a modified version of it that works with both in my answer to the question Write two-dimensional list to JSON file.) Updated Below is a version of my original answer that has been revised several times. Unlike the original, which I posted only to show how to get the first idea in J.F.Sebastian's answer to work, and which like his, returned a non-indented string representation of the object. The latest updated version returns the Python object JSON formatted in isolation. The keys of each coordinate dict will appear in sorted order, as per one of the OP's comments, but only if a sort_keys=True keyword argument is specified in the initial json.dumps() call driving the process, and it no longer changes the object's type to a string along the way. In other words, the actual type of the "wrapped" object is now maintained. I think not understanding the original intent of my post resulted in number of folks downvoting it—so, primarily for that reason, I have "fixed" and improved my answer several times. The current version is a hybrid of my original answer coupled with some of the ideas @Erik Allik used in his answer, plus useful feedback from other users shown in the comments below this answer. The following code appears to work unchanged in both Python 2.7.16 and 3.7.4. from _ctypes import PyObj_FromPtr import json import re class NoIndent(object): """ Value wrapper. """ def __init__(self, value): self.value = value class MyEncoder(json.JSONEncoder): FORMAT_SPEC = '@@{}@@' regex = re.compile(FORMAT_SPEC.format(r'(\d+)')) def __init__(self, **kwargs): # Save copy of any keyword argument values needed for use here. self.__sort_keys = kwargs.get('sort_keys', None) super(MyEncoder, self).__init__(**kwargs) def default(self, obj): return (self.FORMAT_SPEC.format(id(obj)) if isinstance(obj, NoIndent) else super(MyEncoder, self).default(obj)) def encode(self, obj): format_spec = self.FORMAT_SPEC # Local var to expedite access. json_repr = super(MyEncoder, self).encode(obj) # Default JSON. # Replace any marked-up object ids in the JSON repr with the # value returned from the json.dumps() of the corresponding # wrapped Python object. for match in self.regex.finditer(json_repr): # see https://stackoverflow.com/a/15012814/355230 id = int(match.group(1)) no_indent = PyObj_FromPtr(id) json_obj_repr = json.dumps(no_indent.value, sort_keys=self.__sort_keys) # Replace the matched id string with json formatted representation # of the corresponding Python object. json_repr = json_repr.replace( '"{}"'.format(format_spec.format(id)), json_obj_repr) return json_repr if __name__ == '__main__': from string import ascii_lowercase as letters data_structure = { 'layer1': { 'layer2': { 'layer3_1': NoIndent([{"x":1,"y":7}, {"x":0,"y":4}, {"x":5,"y":3}, {"x":6,"y":9}, {k: v for v, k in enumerate(letters)}]), 'layer3_2': 'string', 'layer3_3': NoIndent([{"x":2,"y":8,"z":3}, {"x":1,"y":5,"z":4}, {"x":6,"y":9,"z":8}]), 'layer3_4': NoIndent(list(range(20))), } } } print(json.dumps(data_structure, cls=MyEncoder, sort_keys=True, indent=2)) Output: { "layer1": { "layer2": { "layer3_1": [{"x": 1, "y": 7}, {"x": 0, "y": 4}, {"x": 5, "y": 3}, {"x": 6, "y": 9}, {"a": 0, "b": 1, "c": 2, "d": 3, "e": 4, "f": 5, "g": 6, "h": 7, "i": 8, "j": 9, "k": 10, "l": 11, "m": 12, "n": 13, "o": 14, "p": 15, "q": 16, "r": 17, "s": 18, "t": 19, "u": 20, "v": 21, "w": 22, "x": 23, "y": 24, "z": 25}], "layer3_2": "string", "layer3_3": [{"x": 2, "y": 8, "z": 3}, {"x": 1, "y": 5, "z": 4}, {"x": 6, "y": 9, "z": 8}], "layer3_4": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] } } }


Instruction:
psycopg2 TypeError: not all arguments converted during string formatting

I'm trying execute a simple query, but getting this error no matter how I pass the parameters. Here is the query (I'm using Trac db object to connect to a DB): cursor.execute("""SELECT name FROM "%s".customer WHERE firm_id='%s'""" % (schema, each['id'])) schema and each['id'] both are simple strings print("""SELECT name FROM "%s".customer WHERE firm_id='%s'""" % (schema, each['id'])) Result: SELECT name FROM "Planing".customer WHERE firm_id='135' There is on error is a remove quote after firm_id=, but that way parameter is treated a an integer and ::text leads to the very same error.

Output: It is recommended to not use string interpolation for passing variables in database queries, but using string interpolation to set the table name is fine as long as it's not an external input or you restrict the allowed value. Try: cursor.execute(""" SELECT name FROM %s.customer WHERE firm_id=%%s """ % schema, (each['id'],)) Rules for DB API usage provides guidance for programming against the database.


Instruction:
Jupyter Notebook with Python 3.8 - NotImplementedError

Upgraded recently to Python 3.8, and installed jupyter. However, when trying to run jupyter notebook getting the following error: File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\tornado\platform\asyncio.py", line 99, in add_handler self.asyncio_loop.add_reader(fd, self._handle_events, fd, IOLoop.READ) File "c:\users\user\appdata\local\programs\python\python38\lib\asyncio\events.py", line 501, in add_reader raise NotImplementedError NotImplementedError I know Python 3.8 on windows switched to ProactorEventLoop by default, so I suspect it is related to this. Jupyter does not support Python 3.8 at the moment? Is there a work around?

Output: EDIT This issue exists in older versions of Jupyter Notebook and was fixed in version 6.0.3 (released 2020-01-21). To upgrade to the latest version run: pip install notebook --upgrade Following on this issue through GitHub, it seems the problem is related to the tornado server that jupyter uses. For those that can't wait for an official fix, I was able to get it working by editing the file tornado/platform/asyncio.py, by adding: import sys if sys.platform == 'win32': asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) After the main imports. I expect an official fix for this soon, however.


Instruction:
How to control the parallelism or concurrency of an Airflow installation?

In some of my Apache Airflow installations, DAGs or tasks that are scheduled to run do not run even when the scheduler doesn't appear to be fully loaded. How can I increase the number of DAGs or tasks that can run concurrently? Similarly, if my installation is under high load and I want to limit how quickly my Airflow workers pull queued tasks (such as to reduce resource consumption), what can I adjust to reduce the average load?

Output: Here's an expanded list of configuration options that are available since Airflow v1.10.2. Some can be set on a per-DAG or per-operator basis, but may also fall back to the setup-wide defaults when they are not specified. Options that can be specified on a per-DAG basis: concurrency: the number of task instances allowed to run concurrently across all active runs of the DAG this is set on. Defaults to core.dag_concurrency if not set max_active_runs: maximum number of active runs for this DAG. The scheduler will not create new active DAG runs once this limit is hit. Defaults to core.max_active_runs_per_dag if not set Examples: # Only allow one run of this DAG to be running at any given time dag = DAG('my_dag_id', max_active_runs=1) # Allow a maximum of 10 tasks to be running across a max of 2 active DAG runs dag = DAG('example2', concurrency=10, max_active_runs=2) Options that can be specified on a per-operator basis: pool: the pool to execute the task in. Pools can be used to limit parallelism for only a subset of tasks max_active_tis_per_dag: controls the number of concurrent running task instances across dag_runs per task. Example: t1 = BaseOperator(pool='my_custom_pool', max_active_tis_per_dag=12) Options that are specified across an entire Airflow setup: core.parallelism: maximum number of tasks running across an entire Airflow installation core.dag_concurrency: max number of tasks that can be running per DAG (across multiple DAG runs) core.non_pooled_task_slot_count: number of task slots allocated to tasks not running in a pool core.max_active_runs_per_dag: maximum number of active DAG runs, per DAG scheduler.max_threads: how many threads the scheduler process should use to use to schedule DAGs celery.worker_concurrency: max number of task instances that a worker will process at a time if using CeleryExecutor celery.sync_parallelism: number of processes CeleryExecutor should use to sync task state


Instruction:
Keras AttributeError: 'Sequential' object has no attribute 'predict_classes'

Im attempting to find model performance metrics (F1 score, accuracy, recall) following this guide https://machinelearningmastery.com/how-to-calculate-precision-recall-f1-and-more-for-deep-learning-models/ This exact code was working a few months ago but now returning all sorts of errors, very confusing since i havent changed one character of this code. Maybe a package update has changed things? I fit the sequential model with model.fit, then used model.evaluate to find test accuracy. Now i am attempting to use model.predict_classes to make class predictions (model is a multi-class classifier). Code shown below: model = Sequential() model.add(Dense(24, input_dim=13, activation='relu')) model.add(Dense(18, activation='relu')) model.add(Dense(6, activation='softmax')) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) - history = model.fit(X_train, y_train, batch_size = 256, epochs = 10, verbose = 2, validation_split = 0.2) - score, acc = model.evaluate(X_test, y_test,verbose=2, batch_size= 256) print('test accuracy:', acc) - yhat_classes = model.predict_classes(X_test) last line returns error "AttributeError: 'Sequential' object has no attribute 'predict_classes'" This exact code was working not long ago so struggling a bit, thanks for any help

Output: This function was removed in TensorFlow version 2.6. According to the keras in rstudio reference update to predict_x=model.predict(X_test) classes_x=np.argmax(predict_x,axis=1) Or use TensorFlow 2.5.x . If you are using TensorFlow version 2.5, you will receive the following warning: tensorflow\python\keras\engine\sequential.py:455: UserWarning: model.predict_classes() is deprecated and will be removed after 2021-01-01. Please use instead:* np.argmax(model.predict(x), axis=-1), if your model does multi-class classification (e.g. if it uses a softmax last-layer activation).* (model.predict(x) > 0.5).astype("int32"), if your model does binary classification (e.g. if it uses a sigmoid last-layer activation).


Instruction:
Get hard disk size in Python

I am trying to get the hard drive size and free space using Python (I am using Python 2.7 with macOS). I am trying with os.statvfs('/'), especially with the following code. Is it correct what I am doing? Which definition of the variable giga shall I use? import os def get_machine_storage(): result=os.statvfs('/') block_size=result.f_frsize total_blocks=result.f_blocks free_blocks=result.f_bfree # giga=1024*1024*1024 giga=1000*1000*1000 total_size=total_blocks*block_size/giga free_size=free_blocks*block_size/giga print('total_size = %s' % total_size) print('free_size = %s' % free_size) get_machine_storage() EDIT: statvfs is deprecated in Python 3, do you know any alternative?

Output: For Python 2 till Python 3.3 Note: As a few people mentioned in the comment section, this solution will work for Python 3.3 and above. For Python 2.7 it is best to use the psutil library, which has a disk_usage function, containing information about total, used and free disk space: import psutil hdd = psutil.disk_usage('/') print ("Total: %d GiB" % hdd.total / (2**30)) print ("Used: %d GiB" % hdd.used / (2**30)) print ("Free: %d GiB" % hdd.free / (2**30)) Python 3.3 and above: For Python 3.3 and above, you can use the shutil module, which has a disk_usage function, returning a named tuple with the amounts of total, used and free space in your hard drive. You can call the function as below and get all information about your disk's space: import shutil total, used, free = shutil.disk_usage("/") print("Total: %d GiB" % (total // (2**30))) print("Used: %d GiB" % (used // (2**30))) print("Free: %d GiB" % (free // (2**30))) Output: Total: 931 GiB Used: 29 GiB Free: 902 GiB


Instruction:
OpenCV v1/v2 error: the function is not implemented

I'm trying to get OpenCV working with Python on my Ubuntu machine. I've downloaded and installed OpenCV, but when I attempt to run the following python code (which should capture images from a webcam and push them to the screen) import cv cv.NamedWindow("w1", cv.CV_WINDOW_AUTOSIZE) capture = cv.CaptureFromCAM(0) def repeat(): frame = cv.QueryFrame(capture) cv.ShowImage("w1", frame) time.sleep(10) while True: repeat() I get the following error: The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script So I do what they ask: install the packages, move to the folder from whence I installed OpenCV, and run sudo make uninstall make sudo make install But when I try to run the python, it gives me the same error. Am I missing something?

Output: If it's giving you errors with gtk, try qt. sudo apt-get install libqt4-dev cmake -D WITH_QT=ON .. make sudo make install If this doesn't work, there's an easy way out. sudo apt-get install libopencv-* This will download all the required dependencies(although it seems that you have all the required libraries installed, but still you could try it once). This will probably install OpenCV 2.3.1 (Ubuntu 12.04). But since you have OpenCV 2.4.3 in /usr/local/lib include this path in /etc/ld.so.conf and do ldconfig. So now whenever you use OpenCV, you'd use the latest version. This is not the best way to do it but if you're still having problems with qt or gtk, try this once. This should work. Update - 18th Jun 2019 I got this error on my Ubuntu(18.04.1 LTS) system for openCV 3.4.2, as the method call to cv2.imshow was failing (e.g., at the line of cv2.namedWindow(name) with error: cv2.error: OpenCV(3.4.2). The function is not implemented.). I am using anaconda. Just the below 2 steps helped me resolve: conda remove opencv conda install -c conda-forge opencv=4.1.0 If you are using pip, you can try pip install opencv-contrib-python


Instruction:
How to check if a module is installed in Python and, if not, install it within the code?

I would like to install the modules 'mutagen' and 'gTTS' for my code, but I want to have it so it will install the modules on every computer that doesn't have them, but it won't try to install them if they're already installed. I currently have: def install(package): pip.main(['install', package]) install('mutagen') install('gTTS') from gtts import gTTS from mutagen.mp3 import MP3 However, if you already have the modules, this will just add unnecessary clutter to the start of the program whenever you open it.

Output: EDIT - 2020/02/03 The pip module has updated quite a lot since the time I posted this answer. I've updated the snippet with the proper way to install a missing dependency, which is to use subprocess and pkg_resources, and not pip. To hide the output, you can redirect the subprocess output to devnull: import sys import subprocess import pkg_resources required = {'mutagen', 'gTTS'} installed = {pkg.key for pkg in pkg_resources.working_set} missing = required - installed if missing: python = sys.executable subprocess.check_call([python, '-m', 'pip', 'install', *missing], stdout=subprocess.DEVNULL) Like @zwer mentioned, the above works, although it is not seen as a proper way of packaging your project. To look at this in better depth, read the the page How to package a Python App.


Instruction:
VS Code Python + Black formatter arguments - python.formatting.blackArgs

April 2024 UPDATE, Please read: It appears the answer since late 2023 is the newly accepted answer provided by Mircea, this question is old so older, now incorrect answers, have more upvotes but please scroll down for that answer. original question: I'm using the May 2018 Python extension (released June 2018) for VS Code 1.23.1 on Windows, python 3.6 via Anaconda, conda installing black from conda-forge into my conda environment. In my user settings.json I have the below: "python.formatting.blackArgs": [ "--line-length 80" ], which I'd think would be the correct way to structure this to pass arguments to black in VS Code Python formatting. However, in my python Output pane I get the below: Formatting with black failed. Error: Error: no such option: --line-length 80 If I edit my settings.json to be no args, such as: "python.formatting.blackArgs": [], black works as expected. Does anyone know how to pass arguments correctly to the new (as of June 2018) black formatter?

Output: Hopefully the answer for a more recent VSCode and Black helps: With VsCode Insiders (1.83.0-insider) and Black installed from extensions (v2023.4.1), installed from extensions (https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter) I had to add -l or --line-length and 80 as separate items to File->Preferences->Settings->[type]black->Black-formatter: Args->Add item. In user settings json (Ctrl + Shift + P --> Open User Settings) I have: "black-formatter.args": ["--line-length", "80"] If this doesn't work, there's useful information in the Output Window (you can select Black Formatter) to see the logs from Black.


Instruction:
Remove unnecessary whitespace from Jinja rendered template

I'm using curl to watch the output of my web app. When Flask and Jinja render templates, there's a lot of unnecessary white space in the output. It seems to be added by rendering various components from Flask-WTF and Flask-Bootstrap. I could strip this using sed, but is there a way to control this from Jinja?

Output: Jinja has multiple ways to control whitespace. It does not have a way to prettify output, you have to manually make sure everything looks "nice". The broadest solution is to set trim_blocks and lstrip_blocks on the env. app.jinja_env.trim_blocks = True app.jinja_env.lstrip_blocks = True If you want to keep a newline at the end of the file, set strip_trailing_newlines = False. You can use control characters to modify how the whitespace around a tag works. - always removes whitespace, + always preserves it, overriding the env settings for that tag. The - character can go at the beginning or end (or both) of a tag to control the whitespace in that direction, the + character only makes sense at the beginning of a tag. {%- if ... %} strips before {%- if ... -%} strips before and after {%+ if ... %} preserves before {%+ if ... -%} preserves before and strips after remember that {% endif %} is treated separately The control characters only apply to templates you write. If you include a template or use a macro from a 3rd party, however they wrote the template will apply to that part.


Instruction:
Parameterized queries with psycopg2 / Python DB-API and PostgreSQL

What's the best way to make psycopg2 pass parameterized queries to PostgreSQL? I don't want to write my own escpaing mechanisms or adapters and the psycopg2 source code and examples are difficult to read in a web browser. If I need to switch to something like PyGreSQL or another python pg adapter, that's fine with me. I just want simple parameterization.

Output: psycopg2 follows the rules for DB-API 2.0 (set down in PEP-249). That means you can call execute method from your cursor object and use the pyformat binding style, and it will do the escaping for you. For example, the following should be safe (and work): cursor.execute("SELECT * FROM students WHERE last_name = %(lname)s", {"lname": "Robert'); DROP TABLE students;--"}) Edit: tekHedd's comment rightly points out that the SELECT and the DROP TABLE used different table names, so I fixed it.


Instruction:
Zero pad numpy array

What's the more pythonic way to pad an array with zeros at the end? def pad(A, length): ... A = np.array([1,2,3,4,5]) pad(A, 8) # expected : [1,2,3,4,5,0,0,0] In my real use case, in fact I want to pad an array to the closest multiple of 1024. Ex: 1342 => 2048, 3000 => 3072

Output: For your use case you can use resize() method: A = np.array([1,2,3,4,5]) A.resize(8) This resizes A in place. If there are refs to A numpy throws a vale error because the referenced value would be updated too. To allow this add refcheck=False option. The documentation states that missing values will be 0: Enlarging an array: as above, but missing entries are filled with zeros


Instruction:
Create (sane/safe) filename from any (unsafe) string

I want to create a sane/safe filename (i.e. somewhat readable, no "strange" characters, etc.) from some random Unicode string (which might contain just anything). (It doesn't matter for me whether the function is Cocoa, ObjC, Python, etc.) Of course, there might be infinite many characters which might be strange. Thus, it is not really a solution to have a blacklist and to add more and more to that list over the time. I could have a whitelist. However, I don't really know how to define it. [a-zA-Z0-9 .] is a start but I also want to accept unicode chars which can be displayed in a normal way.

Output: Python: "".join(c for c in filename if c.isalpha() or c.isdigit() or c==' ').rstrip() this accepts Unicode characters but removes line breaks, etc. example: filename = u"ad\nbla'{-+\)(ç?" gives: adblaç edit str.isalnum() does alphanumeric on one step. – comment from queueoverflow below. danodonovan hinted on keeping a dot included. keepcharacters = (' ','.','_') "".join(c for c in filename if c.isalnum() or c in keepcharacters).rstrip()