You can add one or more statuses in settings.py. Scrapy will process requests normally and when one of these statuses is encountered, it will retry that request. You can modify RETRY_HTTP_CODES and add any number of statuses there. You can also control how many times to try with RETRY_TIMES

Show images docker images Show containers docker ps Show running containers docker ps -a Connect to a running container docker exec -it container_id bash Start container docker start container_id Stop container docker stop container_id Remove Container docker rm container_id Pull container docker pull image_name Commit a container docker commit -m "message" -a "Author" container_id username/repo_name Remove all containers docker rm $(docker ps -a -q) Stop all containers docker kill $(docker ps -q) ...

This is one of the tricks on Medium.com that seems to work pretty well. Please use on your own risk. Open Medium.com Find a user with lots of followers Click on link to view those followers Then right click + inspect First include jQuery var jqry = document.createElement('script'); jqry.src = "https://code.jquery.com/jquery-3.3.1.min.js"; document.getElementsByTagName('head')[0].appendChild(jqry); jQuery.noConflict(); Then run the following snippet to follow. It is best practice to not follow more than 120 people per day. $('.js-followButton').each(function(index, item) { $(item).trigger("click"); console.log('Followed ' + index + '/ 120'); if (index > 100) { return false; } });

// iterate over a list $.each(list, function(key, field) { // something here }); // find an element across the page and iterate over each $(document).find('item').each(function(item, object) { // something here });

I often encounter this case where I land on a website that has something I’d really like to read. However, the white text on black background and 10px font size makes me want to throw up. Hence, I’m leaving this tiny piece here to fix that situation. Once you run this piece of code, it will do following things. Increase website font size Change background color to white Change text color to black If website doesn’t have jQuery, first run this ...

jQuery as we know comes loaded with most websites today. However, if there is a case when you’d like to run some experiments on website using jQuery but it isn’t being load by default, you can run following command var jqry = document.createElement('script'); jqry.src = "[https://code.jquery.com/jquery-3.3.1.min.js";](https://code.jquery.com/jquery-3.3.1.min.js) document.getElementsByTagName('head')\[0\].appendChild(jqry); jQuery.noConflict();

$("html, body").animate({ scrollTop: $(document).height()-$(window).height() });

Following is a sample Ajax template that can be used when you’re not in the mood of surfing through docs $.ajax({ url: "http://google.com", type: "post", dataType: "json", data: {name: item, value: value}, success: function (response) { // do somethinig here }, error: function(jqXHR, textStatus, errorThrown) { console.log(textStatus, errorThrown); } });

$.each iterates over all elements specified and performs your code Iterate over a list $.each(list, function(key, field) { // something here }); Find an element across the page and iterate over each $(document).find('item').each(function(item, object) { // something here });

YouTube-dl Download and Documentation I was a long time user of YTMusic, a premium music subscription service. One day they randomly stopped accepting payments from my card. I tried a few others but nothing worked. In a few days I lost access to all my songs and I had to start with a new account with no explanation for what happened. I decided to not let this happen again and started building a backup of songs. Every time I liked a song, I would append it to list on Evernote. Now whenever I need to have all those songs available locally, I run the following command which iterates over each song title, searches youtube and dowloads first video’s audio. This is about 96% accurate :D ...

The following snippet can be used for filtering out URLs that where keywords match title or url. They are then deleted from history saving you lots of time that you’d otherwise spend manually filtering and deleting. Google Chrome must be closed before running this script. import sqlite3, webbrowser def cleanChrome(): con = sqlite3.connect('C:\\Users\\{username}\\AppData\\Local\\Google\\Chrome\\User Data\\Default\\History') cursor = con.cursor() cursor.execute("select id, url, title from urls") urls = cursor.fetchall() # keywords that you'd like to detect and delete # it looks for these keywords in page title and urls keywords = [] total = len(urls) done = 0 deleted = 0 pendingIds = [] for url in urls: jenny.output(f'Processing {done} / {total} urls [deleted: {deleted}]') uid = url[0] link = url[1].lower() title = url[2].lower() for keyword in keywords: if keyword in link or keyword in title: jenny.output(f'{keyword} matched, deleting..') pendingIds.append((uid,)) deleted += 1 done += 1 query = 'DELETE FROM urls WHERE id=?' cursor.executemany(query, pendingIds) con.commit()

with open(path, mode='r') as csvFile: reader = csv.DictReader(csvFile) lineCount = 0 for row in reader: if lineCount > 0: # do stuff here lineCount += 1

First, make sure virtualenv installed or install it by running pip install virtualenv After this you can simply run virtualenv name_of_env_to_create You can activate it by going to following directory where you ran the command ./env/Scripts/activate.bat

Scrapy spider can close unexpectedly for many reasons. If you’d like to notify yourself or do anything whenever a spider closes (expectedly or unexpectedly) Create a function named anything e.g crawlFinished() Then paste self.crawlFinish() at the bottom of closed() function Now your function will be executed each time crawler exits

function compare($startDate, $endDate) { $weekends = 0; $startDate = new DateTime($startDate); $endDate = new DateTime($endDate); while($startDate <= $endDate ) { $timestamp = strtotime($startDate->format('d-m-Y')); $currentDay = strtolower(date('D', $timestamp)); if (!$weekends && $currentDay == 'sun') { $weekends += 1; } elseif ($currentDay == 'sat') { $weekends += 1; } $startDate->modify('+1 day'); } return $weekends; }

import webbrowser webbrowser.register('chrome', None, webbrowser.BackgroundBrowser("C://Program Files (x86)//Google//Chrome//Application//chrome.exe")) webbrowser.get('chrome').open('http://url.com')

YouTube-dl Download and Documentation This command iterates over entire channel and downloads each video one by one. youtube-dl -f best -ciw -o "%(title)s.%(ext)s" -v url_of_channel

YouTube-dl Download and Documentation Download the best quality audio in mp3 format using the command below. This command also utilises ffmpeg to extract audio. In general, it is a good idea to have ffmpeg installed on your system if you plan to use YouTube-dl youtube-dl -f bestaudio --extract-audio --audio-format mp3 --audio-quality 0 'url_here'

from scrapy.shell import inspect_response inspect_response(response, self) Read this for more details

response.xpath("//*[contains(text(), 'txt goes here')]").getall()

If Laravel is not installed, first run composer create-project laravel/laravel composer require laravel/ui php artisan ui bootstrap --auth

I was building an API. I had already used auth:api middleware. I was sure that a user was logged in. However, I wanted to protect some of my routes further by only allowing an admin to access them. I was using level field in users table. A level 1 meant it was an admin. Hence, I created an Admin middleware to do what I had in mind. Here is how it can be done. ...

There are some cases when you need to add your own custom functions in Laravel. I often have my own custom helpers for speeding up development. You can easily add your own files by following this quick guide. Create a folder in app\Http folder. Let’s assume you created Helpers You can place any number of php files in this directory. Then open composer.json found in your root directory and find "autoload-dev": { "psr-4": { "Tests\\": "tests/" } } Then replace it so it looks like this "autoload-dev": { "psr-4": { "Tests\\": "tests/" }, "files": [ "app/Helpers/filename.php" ] } Finally, run composer dump-autoload and you are good to go

Follow command can be used to create all three in one line php artisan make:model Modelname -mcr

If you are using sublime text run: Ctrl + P: AppServiceProvider.php Then at start of file paste use Illuminate\Support\Facades\Schema; Paste following code in boot() function Schema::defaultStringLength(191);

You can run the migrations and seeders in the same line like this php artisan migrate:fresh --seed

A Laravel project ideally has many tests depending on the size of the project. We don’t want one test’s results to impact others. This is where RefreshDatabase trait comes in handy. It resets database after each test to make sure you get the clean slate for your next test. You can use it like this. First, paste this at top of your test file use Illuminate\Foundation\Testing\RefreshDatabase; Then right after opening { of your test class, paste following use RefreshDatabase; ...

There are two main ways to use proxies in Scrapy. You can either use it per request basis or use it with every scrapy outgoing request. How to use proxy with a single request proxy = 'proxy_here' return Request(url=url, callback=self.parse, meta={"proxy": proxy}) How to use proxy with every request Go to to middlewares.py and update process_request method and paste following code proxy = 'proxy_here' request.meta['proxy'] = proxy

Here is a quick snippet that showed how classes are loaded in BriskLimbs, a video sharing project of mine spl_autoload_register(function ($class) { $class = ucfirst($class); if (file_exists(MODEL_DIRECTORY . '/' . $class . '.php')) { require_once MODEL_DIRECTORY . '/' . $class . '.php'; } else { $class = strtolower($class); if (file_exists(HELPERS_DIRECTORY . '/' . $class . '.php')) { require_once HELPERS_DIRECTORY . '/' . $class . '.php'; } else { exit(MODEL_DIRECTORY . '/' . $class . '.php'); exit('Error Loading Class File |' . $class . '| core:' . CORE_DIRECTORY); } } });

Laravel has a validated function which runs your defined rules and automatically sends errors to your view. This is awesome for most projects but sometimes you may want to handle / read these errors yourself. Or maybe just do it before sending them to view. Here is how you can do that. First of all, add validator at top of your file use Illuminate\Support\Facades\Validator; Then create validator with your rules like this $validator = Validator::make($requestData, [ 'user_id' => ['required', 'integer'] ]); Then use following snippet to get errors as array if ($validator->fails()) { $fieldsWithErrors = $validator->messages()->get('*'); // do anything here }