[phpBB Debug] PHP Notice: in file [ROOT]/includes/functions_content.php on line 77: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/functions_content.php on line 77: Undefined variable: Array
[phpBB Debug] PHP Notice: in file [ROOT]/includes/functions_content.php on line 77: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/functions_content.php on line 77: Undefined variable: Array
[phpBB Debug] PHP Notice: in file [ROOT]/includes/functions_content.php on line 77: Array to string conversion
[phpBB Debug] PHP Notice: in file [ROOT]/includes/functions_content.php on line 77: Undefined variable: Array
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4781: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3907)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4783: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3907)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4784: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3907)
[phpBB Debug] PHP Warning: in file [ROOT]/includes/functions.php on line 4785: Cannot modify header information - headers already sent by (output started at [ROOT]/includes/functions.php:3907)
forum.blocklist.de • View topic - Blocklist.de & NGINX

  • Advertisement

Blocklist.de & NGINX

Anleitungen zu Fail2Ban, blocklist.de und x-arf

Blocklist.de & NGINX

Postby VBTECH » 30. Apr 2014, 09:01

Hi,

ich habe ein Script für den Einbau der Spamhaus-Listen (DROP, EDROP) in NGINX gefunden ().

Das funktioniert für die beiden genannten Listen auch perfekt, nur ist es mir bisher nicht gelungen, auch die gesperrten IPs von Blocklist.de einzubinden. NGINX startet dann nicht mehr neu :| , sondern verlangt ein ";" bzw. "or" am Ende.

Versucht hab ich es wie folgt:

## The lists URIs and respective filenames.
DROP_URI=http://www.spamhaus.org/drop/drop.txt
DROP_FILENAME=$(basename $DROP_URI)
EDROP_URI=http://www.spamhaus.org/drop/edrop.txt
EDROP_FILENAME=$(basename $EDROP_URI)
BLOCKLIST_URI= http://lists.blocklist.de/lists/all.txt
BLOCKLIST_FILENAME=$(basename $BLOCKLIST_URI)


## If the file doesn't exist create it and process the lists.
if [ ! -f $OUTPUT_FILE ]; then
process_lists $OUTPUT_FILE $DROP_FILENAME $DROP_URI
print_added_lines $DROP_FILENAME
process_lists $OUTPUT_FILE $EDROP_FILENAME $EDROP_URI "append"
print_added_lines $EDROP_FILENAME
process_lists $OUTPUT_FILE $BLOCKLIST_FILENAME $BLOCKLIST_URI "append"
print_added_lines $BLOCKLIST_FILENAME

WRITTEN_FILE="yes"

## Grab the drop files if the Expire date of the current list has
## passed already.
if [ "$(get_list_expire_date $DROP_FILENAME)" -lt "$(date '+%s')" ]; then
process_lists $OUTPUT_FILE $DROP_FILENAME $DROP_URI
print_added_lines $DROP_FILENAME
WRITTEN_FILE="yes"
fi

## Ditto for the extended drop list.
if [ "$(get_list_expire_date $EDROP_FILENAME)" -lt "$(date '+%s')" ]; then
process_lists $OUTPUT_FILE $EDROP_FILENAME $EDROP_URI "append"
print_added_lines $EDROP_FILENAME
WRITTEN_FILE="yes"
fi

## Ditto for the extended blocklist.
if [ "$(get_list_expire_date $BLOCKLIST_FILENAME)" -lt "$(date '+%s')" ]; then
process_lists $OUTPUT_FILE $BLOCKLIST_FILENAME $BLOCKLIST_URI "append"
print_added_lines $BLOCKLIST_FILENAME
WRITTEN_FILE="yes"
fi


Hat jemand eine Idee, dass NGINX nicht meckert. Was muss ich noch einfügen?
VBTECH
 
Posts: 32
Joined: 26. Mar 2013, 15:54

Re: Blocklist.de & NGINX

Postby VBTECH » 30. Apr 2014, 09:02

PS.: Den Autor des Scripts hab ich angeschrieben, leider antwortet er nicht ... :(
VBTECH
 
Posts: 32
Joined: 26. Mar 2013, 15:54

Re: Blocklist.de & NGINX

Postby Martin » 30. Apr 2014, 09:27

Hi,

wie lautet denn die genaue Fehlermeldung?
Mfg Martin
http://www.blocklist.de/de/ Fail2Ban Reporting Service
User avatar
Martin
 
Posts: 397
Joined: 14. Sep 2010, 11:54

Re: Blocklist.de & NGINX

Postby VBTECH » 30. Apr 2014, 11:29

Hi,

zunächst importiert er die Listen

#File drop.txt adding 583 networks
File edrop.txt adding 22 networks
File ftp.txt adding 123 networks
File /etc/nginx/drop_list.conf written.

ftp.txt ist die Blocklist für die FTP-Angriffe (hier mal testweise ausgewählt)

dann NGINX restart ergibt:

nginx: [emerg] unexpected end of file, expecting ";" or "}" in /etc/nginx/drop_list.conf:738
nginx: configuration file /etc/nginx/nginx.conf test failed

Zeile 738 ist das Ende der importierten Liste von Blocklist.de

GEO in NGINX verlangt, dass am Ende der IP eine "1" steht, gefolgt von ";", z.B. so

5.34.242.0/24 1; # SBL154880
VBTECH
 
Posts: 32
Joined: 26. Mar 2013, 15:54

Re: Blocklist.de & NGINX

Postby VBTECH » 4. May 2014, 10:12

Hat niemand eine Idee, was man ändern/ergänzen müsste?

Bin (leider) kein IT-Techniker, bekomm es also nicht selbst hin . . .

Das original des Scripts sieht so aus (meine Änderungen, im Beitrag oben drüber); Am Ende müsste für nginx geo

IP 1; rauskommen



#!/bin/bash

### Simple shell script to create a list of IPs suitable for the Nginx
### geo module http://nginx.org/en/docs/http/ngx_http_geo_module.html
### based on the Spamhaus DROP and EDROP lists. See
### http://www.spamhaus.org/drop.

SCRIPTNAME=${0##*/}
CURL=$(command -v curl) || exit 0

## The lists URIs and respective filenames.
DROP_URI=http://www.spamhaus.org/drop/drop.txt
DROP_FILENAME=$(basename $DROP_URI)
EDROP_URI=http://www.spamhaus.org/drop/edrop.txt
EDROP_FILENAME=$(basename $EDROP_URI)

function print_usage() {
echo "Usage: $SCRIPTNAME [output file]"
}

## Get the drop list using cURL.
## $1: the list URI.
function get_drop_list() {
$CURL -s -O $1
}

## Get the expire date in UNIX epoch format.
## $1: the list file.
function get_list_expire_date() {
echo $(date --date="$(sed -n 's/^.*Expires:\(.*\)$/\1/p' $1)" '+%s')
}

## Get the list date.
## $1: the list file.
function get_list_date() {
echo $(date --date="$(sed -n 's/^.*Last-Modified:\(.*\)$/\1/p' $1)" '+%s')
}

## Format the list according to Nginx geo module format.
## $1: the list file.
function format_geo_ips() {
sed -e '/^;/d' -e 's/;/1; #/g' $1
}

## Get the list headers. Last-Modified and Expires date.
## $1: the list file.
function print_list_headers() {
sed -n 's/^; \(.*\)$/# \1/p' $1
}

## Process each list.
## $1: output filename.
## $2: list filename.
## $3: list URI.
function process_lists() {
get_drop_list $3
## Write the DROP/EDROP list file for Nginx geo.
if [ "$4" = "append" ]; then
print_list_headers $2 >> $1
else
print_list_headers $2 > $1
fi
format_geo_ips $2 >> $1
}

## Echo the number of added lines to a particular file.
## #1: drop list file.
function print_added_lines() {
echo $(wc -l $1) | awk '{print "File " $2 " adding " $1 - 3 " networks"}'
}

## Check the number of arguments.
if [ $# -gt 1 ]; then
print_usage
exit 1
fi

## Get the output filename.
OUTPUT_FILE=${1-/etc/nginx/drop_list.conf}
WRITTEN_FILE="no"

## If the file doesn't exist create it and process the lists.
if [ ! -f $OUTPUT_FILE ]; then
process_lists $OUTPUT_FILE $DROP_FILENAME $DROP_URI
print_added_lines $DROP_FILENAME
process_lists $OUTPUT_FILE $EDROP_FILENAME $EDROP_URI "append"
print_added_lines $EDROP_FILENAME
WRITTEN_FILE="yes"
fi

## Grab the drop files if the Expire date of the current list has
## passed already.
if [ "$(get_list_expire_date $DROP_FILENAME)" -lt "$(date '+%s')" ]; then
process_lists $OUTPUT_FILE $DROP_FILENAME $DROP_URI
print_added_lines $DROP_FILENAME
WRITTEN_FILE="yes"
fi

## Ditto for the extended drop list.
if [ "$(get_list_expire_date $EDROP_FILENAME)" -lt "$(date '+%s')" ]; then
process_lists $OUTPUT_FILE $EDROP_FILENAME $EDROP_URI "append"
print_added_lines $EDROP_FILENAME
WRITTEN_FILE="yes"
fi

## Write a message about the writing of the file.
if [ "$WRITTEN_FILE" = "yes" ]; then
echo "File $OUTPUT_FILE written."
fi
VBTECH
 
Posts: 32
Joined: 26. Mar 2013, 15:54

Re: Blocklist.de & NGINX

Postby Virus2500@me.com » 5. May 2014, 23:04

Virus2500@me.com
 
Posts: 8
Joined: 26. May 2011, 01:54

Re: Blocklist.de & NGINX

Postby VBTECH » 6. May 2014, 04:54

Hi,

zunächst Danke!

Habs mit ftp.txt, imap.txt und all.txt getestet und bekomme leider folgende (identische) Fehlermeldungen:

# bash nginx-drop-fetch
sed: kann imap.txt nicht lesen: Datei oder Verzeichnis nicht gefunden
File imap.txt adding 838 networks
File /etc/nginx/drop_list.conf written.

# nginx restart
nginx: [emerg] invalid number of the geo parameters in /etc/nginx/drop_list.conf:1476
nginx: configuration file /etc/nginx/nginx.conf test failed

Ein Blick in die drop_lis.conf zeigt folgendes:

Die Listen von blocklist.de werden (im Block) jeweils 2x eingelesen, zuerst einmal ohne "1;" und danach einmal mit "1;" am Ende der Zeile.
Bei den IPs (im Block) mit "1;" am Ende fehlt zwischendurch mal plötzlich "1;" am Ende. Nach Vergleich mit dem Original, z.B. der all.txt von Blocklist.de, scheinen da IPv6-Adressen drin zu stehen, die das Script nicht verarbeiten kann
Die letzte IP des Blocks mit "1;" am Ende wird ohne "1;" dargestellt.
VBTECH
 
Posts: 32
Joined: 26. Mar 2013, 15:54

Re: Blocklist.de & NGINX

Postby Virus2500@me.com » 6. May 2014, 11:46

Hello,

aendere doch bitte mal in der function_format_nogeo_ips
sed ':a;N;$!ba;s/\n/ 1;\n/g' $1
auf
sed -e 's/$/ 1;/g' $1

Du müsstest allerdings die /etc/nginx/drop_list mal leeren und auch die all.txt, edrop.txt und drop.txt löschen und nochmal starten.

Selbst dann gibt es allerdings noch immer ein Problem weil die edrop und drop liste ein ablaufdatum haben. Die all.txt von blocklist aber nicht und irgendwo checkt das script es dann nicht und schreibt die IPs von blocklist immer wieder neu rein. Sprich bei jedem run des scripts kommt die Liste wieder rein. Mit Bash scripten hab ich zu wenig erfahrung um das umgehen zu können.

Wenn es darum geht das die IPs von blocklist gar keinen Zugriff haben kann ich dir mein Perl script anbieten (ich weiß eigenwerbung stinkt :) ).
https://github.com/virus2500/blocklist-with-ipset
Das arbeitet mit Perl, iptables und ipset.

Wenn du es aber direkt mit nginx blocken willst steh ich wohl leider jetzt auch an... :(
Virus2500@me.com
 
Posts: 8
Joined: 26. May 2011, 01:54

Re: Blocklist.de & NGINX

Postby VBTECH » 8. May 2014, 10:48

Also ich habs probiert:

Er schreibt jetzt die IPs von z.B. ftp..txt alle mit "1;" am Ende, allerdings 2x in die Datei.

erst oim Block ohne "1;" und dann mit "1;"

Du musst ein wenig zurück gegehnin der Datei, dann siehst du es.

Da muss irgendwo was doppelt im Script sein . . . ich find es aber nicht
VBTECH
 
Posts: 32
Joined: 26. Mar 2013, 15:54

Re: Blocklist.de & NGINX

Postby Virus2500@me.com » 8. May 2014, 11:44

Virus2500@me.com
 
Posts: 8
Joined: 26. May 2011, 01:54

Re: Blocklist.de & NGINX

Postby VBTECH » 13. May 2014, 05:36

Hi,

ich habs getest (vorher alle alten Dateien zu dem Script gelöscht) und wieder sind die IPs von blocklist.de doppelt drin und zwar unmittelbar nach den IPs von Spamhaus (EDROP) und ohne "; 1":

208.38.131.0/24 1; # SBL178294
208.38.135.0/24 1; # SBL178295
212.95.144.0/22 1; # SBL198435
1.163.161.184
1.164.88.178
1.165.169.77

Danach kommen die IPs von blocklist.de erneut, aber in dieser Form:

46.227.71.217 1;
46.227.71.218 1;
46.227.71.219 1;
46.227.71.220 1;
46.227.71.221 1;

:(

Ich versteh es nicht mehr ...
VBTECH
 
Posts: 32
Joined: 26. Mar 2013, 15:54

Re: Blocklist.de & NGINX

Postby Virus2500@me.com » 13. May 2014, 07:13

Hast du es nochmal mit dem ganzen Script probiert was ich in viewtopic.php?f=11&t=210&p=734#p731 gepostet habe? Vielleicht hab ich wirklich irgendwas anders als du?!

Ansonsten versteh ich auch nicht wo der Unterschied wäre?!
Virus2500@me.com
 
Posts: 8
Joined: 26. May 2011, 01:54

Re: Blocklist.de & NGINX

Postby VBTECH » 13. May 2014, 08:44

Hi,

ja, ich habe das sogar rauskopiert und auf dem Server einkopiert! Ich versteh es nicht, was da los ist, schon beim Erstellen von drop.list sihet man, dass die Blocklist.de IPs doppelt geladen werden.

Gibt es hier noch jemanden der uns weiterhelfen kann ....????
VBTECH
 
Posts: 32
Joined: 26. Mar 2013, 15:54

Re: Blocklist.de & NGINX

Postby Martin » 17. May 2014, 20:43

Hi,

auf der Shell könnte man mit:
cat drop.list | sort | uniq > datei-einzeln.list
sortiert und doppelte IPs aussortiert werden.
Mfg Martin
http://www.blocklist.de/de/ Fail2Ban Reporting Service
User avatar
Martin
 
Posts: 397
Joined: 14. Sep 2010, 11:54


Return to Anleitungen

Who is online

Users browsing this forum: Bing [Bot] and 1 guest

  • Advertisement
cron
figurative