h4cker/osint/README.md

# Open-source Intelligence (OSINT)

Open-source intelligence (OSINT) is data collected from open source and publicly available sources. The following are a few OSINT resources and references:

## Passive Recon Tools:
- [AMass](https://github.com/OWASP/Amass)
- [Exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/)
- [ExtractMetadata](http://www.extractmetadata.com)
- [Findsubdomains](https://findsubdomains.com/)
- [FOCA](https://elevenpaths.com)
- [IntelTechniques](https://inteltechniques.com)
- [Maltego](https://www.paterva.com/web7/)
- [Recon-NG](https://github.com/lanmaster53/recon-ng)
- [Scrapy](https://scrapy.org)
- [Screaming Frog](https://www.screamingfrog.co.uk)
- [Shodan](https://shodan.io)
- [SpiderFoot](http://spiderfoot.net)
- [theHarvester](https://github.com/laramies/theHarvester)
- [Visual SEO Studio](https://visual-seo.com/)
- [Web Data Extractor](http://www.webextractor.com)
- [Xenu](http://home.snafu.de)
- [ParamSpider](https://github.com/devanshbatham/ParamSpider)


## Open Source Threat Intelligence

- [GOSINT](https://github.com/ciscocsirt/gosint) - a project used for collecting, processing, and exporting high quality indicators of compromise (IOCs). GOSINT allows a security analyst to collect and standardize structured and unstructured threat intelligence.
- [Awesome Threat Intelligence](https://github.com/santosomar/awesome-threat-intelligence) - A curated list of awesome Threat Intelligence resources. This is a great resource and I try to contribute to it.


### Website Exploration and "Google Hacking"
- censys : https://censys.io
- Certficate Search: https://crt.sh/
- ExifTool: https://www.sno.phy.queensu.ca/~phil/exiftool
- Google Hacking Database (GHDB): https://www.exploit-db.com/google-hacking-database
- Google Transparency Report: https://transparencyreport.google.com/https/certificates
- Huge TLS/SSL certificate DB with advanced search: https://certdb.com
- netcraft: https://searchdns.netcraft.com
- SiteDigger: http://www.mcafee.com/us/downloads/free-tools/sitedigger.aspx
- Spyse: https://spyse.com

### Data Breach Query Tools
- BaseQuery: https://github.com/g666gle/BaseQuery
- Buster: https://github.com/sham00n/buster
- h8mail: https://github.com/khast3x/h8mail
- LeakLooker: https://github.com/woj-ciech/LeakLooker
- PwnDB: https://github.com/davidtavarez/pwndb
- Scavenger: https://github.com/rndinfosecguy/Scavenger
- WhatBreach: https://github.com/Ekultek/WhatBreach

### IP address and DNS Lookup Tools
- [bgp](https://bgp.he.net/)
- [Bgpview](https://bgpview.io/)
- [DataSploit (IP Address Modules)](https://github.com/DataSploit/datasploit/tree/master/ip)
- [Domain Dossier](https://centralops.net/co/domaindossier.aspx)
- [Domaintoipconverter](http://domaintoipconverter.com/) 
- [Googleapps Dig](https://toolbox.googleapps.com/apps/dig/)
- [Hurricane Electric BGP Toolkit](https://bgp.he.net/)
- [ICANN Whois](https://whois.icann.org/en)
- [Massdns](https://github.com/blechschmidt/massdns) 
- [Mxtoolbox](https://mxtoolbox.com/BulkLookup.aspx)
- [Ultratools ipv6Info](https://www.ultratools.com/tools/ipv6Info)
- [Viewdns](https://viewdns.info/) 
- [Umbrella (OpenDNS) Popularity List](http://s3-us-west-1.amazonaws.com/umbrella-static/index.html) 

### Social Media
* [A tool to scrape LinkedIn](https://github.com/dchrastil/TTSL)
* [cree.py](https://github.com/ilektrojohn/creepy)

### Acquisitions and 
- [OCCRP Aleph](https://aleph.occrp.org/) - The global archive of research material for investigative reporting.
### Whois
WHOIS information is based upon a tree hierarchy. ICANN (IANA) is the authoritative registry for all of the TLDs and is a great starting point for all manual WHOIS queries.

- ICANN: http://www.icann.org
- IANA: http://www.iana.com
- NRO: http://www.nro.net
- AFRINIC: http://www.afrinic.net
- APNIC: http://www.apnic.net
- ARIN: http://ws.arin.net
- LACNIC: http://www.lacnic.net
- RIPE: http://www.ripe.net

### BGP looking glasses
- BGP4: http://www.bgp4.as/looking-glasses
- BPG6: http://lg.he.net/

### DNS
- dnsenum -	https://code.google.com/p/dnsenum
- dnsmap: https://code.google.com/p/dnsmap
- dnsrecon: https://www.darkoperator.com/tools-and-scripts
- dnstracer: https://www.mavetju.org/unix/dnstracer.php
- dnswalk: https://sourceforge.net/projects/dnswalk

## The OSINT Framework
- [OSINT Framework](https://osintframework.com)


## Dark Web OSINT Tools
### Dark Web Search Engine Tools
- [Ahmia Search Engine](https://ahmia.fi) and [their GitHub repo](https://github.com/ahmia/ahmia-site)
- [DarkSearch](https://darksearch.io) and their [GitHub repo](https://github.com/thehappydinoa/DarkSearch)
- [Katana](https://github.com/adnane-X-tebbaa/Katana)
- [OnionSearch](https://github.com/megadose/OnionSearch)
- [Search Engines for Academic Research](https://www.itseducation.asia/deep-web.htm)
- [DarkDump](https://github.com/josh0xA/darkdump)

### Tools to Obtain Information of .onion Links
- [H-Indexer](http://jncyepk6zbnosf4p.onion/onions.html)
- [Hunchly](https://www.hunch.ly/darkweb-osint)
- [Tor66 Fresh Onions](http://tor66sewebgixwhcqfnp5inzp5x5uohhdy3kvtnyfxc2e5mxiuh34iid.onion/fresh)

### Tools to scan onion links
- [Onioff](https://github.com/k4m4/onioff)
- [Onion-nmap](https://github.com/milesrichardson/docker-onion-nmap)
- [Onionscan](https://github.com/s-rah/onionscan)

### Tools to Crawl Dark Web Data
- [TorBot](https://github.com/DedSecInside/TorBot)
- [TorCrawl](https://github.com/MikeMeliz/TorCrawl.py)
- [OnionIngestor](https://github.com/danieleperera/OnionIngestor)

### Other Great Intelligence Gathering Sources and Tools
- Resources from Pentest-standard.org - http://www.pentest-standard.org/index.php/PTES_Technical_Guidelines#Intelligence_Gathering

### Active Recon
- Tons of references to scanners and vulnerability management software for active reconnaissance - http://www.pentest-standard.org/index.php/PTES_Technical_Guidelines#Vulnerability_Analysis
Update README.md 2021-02-15 01:27:18 +00:00			`# Open-source Intelligence (OSINT)`
adding OSINT resources 2018-01-18 03:45:47 +00:00
			`Open-source intelligence (OSINT) is data collected from open source and publicly available sources. The following are a few OSINT resources and references:`

Update README.md 2020-05-22 05:45:22 +00:00			`## Passive Recon Tools:`
Update README.md 2020-08-27 00:09:25 +00:00			`- [AMass](https://github.com/OWASP/Amass)`
Update README.md 2020-07-29 01:17:17 +00:00			`- [Exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/)`
			`- [ExtractMetadata](http://www.extractmetadata.com)`
			`- [Findsubdomains](https://findsubdomains.com/)`
			`- [FOCA](https://elevenpaths.com)`
			`- [IntelTechniques](https://inteltechniques.com)`
Update README.md 2018-08-07 20:33:50 +00:00			`- [Maltego](https://www.paterva.com/web7/)`
Update README.md 2019-11-14 18:05:20 +00:00			`- [Recon-NG](https://github.com/lanmaster53/recon-ng)`
Update README.md 2018-08-07 20:33:50 +00:00			`- [Scrapy](https://scrapy.org)`
			`- [Screaming Frog](https://www.screamingfrog.co.uk)`
Update README.md 2020-07-29 01:17:17 +00:00			`- [Shodan](https://shodan.io)`
			`- [SpiderFoot](http://spiderfoot.net)`
			`- [theHarvester](https://github.com/laramies/theHarvester)`
			`- [Visual SEO Studio](https://visual-seo.com/)`
Adding other OSINT tools 2018-08-07 20:40:04 +00:00			`- [Web Data Extractor](http://www.webextractor.com)`
Update README.md 2020-07-29 01:17:17 +00:00			`- [Xenu](http://home.snafu.de)`
adding OSINT tools 2021-02-24 01:29:10 +00:00			`- [ParamSpider](https://github.com/devanshbatham/ParamSpider)`
Update README.md 2020-05-22 05:45:22 +00:00
Update README.md 2021-03-18 18:51:48 +00:00
Update README.md 2020-05-22 05:45:22 +00:00			`## Open Source Threat Intelligence`

			`- [GOSINT](https://github.com/ciscocsirt/gosint) - a project used for collecting, processing, and exporting high quality indicators of compromise (IOCs). GOSINT allows a security analyst to collect and standardize structured and unstructured threat intelligence.`
			`- [Awesome Threat Intelligence](https://github.com/santosomar/awesome-threat-intelligence) - A curated list of awesome Threat Intelligence resources. This is a great resource and I try to contribute to it.`


Adding breached data search tools 2021-06-18 22:48:37 +00:00			`### Website Exploration and "Google Hacking"`
			`- censys : https://censys.io`
			`- Certficate Search: https://crt.sh/`
			`- ExifTool: https://www.sno.phy.queensu.ca/~phil/exiftool`
			`- Google Hacking Database (GHDB): https://www.exploit-db.com/google-hacking-database`
			`- Google Transparency Report: https://transparencyreport.google.com/https/certificates`
			`- Huge TLS/SSL certificate DB with advanced search: https://certdb.com`
			`- netcraft: https://searchdns.netcraft.com`
			`- SiteDigger: http://www.mcafee.com/us/downloads/free-tools/sitedigger.aspx`
			`- Spyse: https://spyse.com`
Update README.md 2020-08-11 19:28:55 +00:00
Adding breached data search tools 2021-06-18 22:48:37 +00:00			`### Data Breach Query Tools`
			`- BaseQuery: https://github.com/g666gle/BaseQuery`
			`- Buster: https://github.com/sham00n/buster`
			`- h8mail: https://github.com/khast3x/h8mail`
			`- LeakLooker: https://github.com/woj-ciech/LeakLooker`
			`- PwnDB: https://github.com/davidtavarez/pwndb`
			`- Scavenger: https://github.com/rndinfosecguy/Scavenger`
			`- WhatBreach: https://github.com/Ekultek/WhatBreach`
Update README.md 2020-08-11 19:28:55 +00:00
			`### IP address and DNS Lookup Tools`
Update README.md 2020-05-22 05:45:22 +00:00			`- [bgp](https://bgp.he.net/)`
			`- [Bgpview](https://bgpview.io/)`
			`- [DataSploit (IP Address Modules)](https://github.com/DataSploit/datasploit/tree/master/ip)`
			`- [Domain Dossier](https://centralops.net/co/domaindossier.aspx)`
			`- [Domaintoipconverter](http://domaintoipconverter.com/)`
			`- [Googleapps Dig](https://toolbox.googleapps.com/apps/dig/)`
			`- [Hurricane Electric BGP Toolkit](https://bgp.he.net/)`
			`- [ICANN Whois](https://whois.icann.org/en)`
			`- [Massdns](https://github.com/blechschmidt/massdns)`
			`- [Mxtoolbox](https://mxtoolbox.com/BulkLookup.aspx)`
			`- [Ultratools ipv6Info](https://www.ultratools.com/tools/ipv6Info)`
			`- [Viewdns](https://viewdns.info/)`
			`- [Umbrella (OpenDNS) Popularity List](http://s3-us-west-1.amazonaws.com/umbrella-static/index.html)`

Adding breached data search tools 2021-06-18 22:48:37 +00:00			`### Social Media`
Update README.md 2023-08-02 02:53:13 +00:00			`* [A tool to scrape LinkedIn](https://github.com/dchrastil/TTSL)`
			`* [cree.py](https://github.com/ilektrojohn/creepy)`
Update README.md 2020-08-11 19:28:55 +00:00
Update README.md 2023-08-02 02:53:13 +00:00			`### Acquisitions and`
			`- [OCCRP Aleph](https://aleph.occrp.org/) - The global archive of research material for investigative reporting.`
Adding breached data search tools 2021-06-18 22:48:37 +00:00			`### Whois`
Update README.md 2020-08-11 19:28:55 +00:00			`WHOIS information is based upon a tree hierarchy. ICANN (IANA) is the authoritative registry for all of the TLDs and is a great starting point for all manual WHOIS queries.`
Adding breached data search tools 2021-06-18 22:48:37 +00:00
			`- ICANN: http://www.icann.org`
			`- IANA: http://www.iana.com`
			`- NRO: http://www.nro.net`
			`- AFRINIC: http://www.afrinic.net`
			`- APNIC: http://www.apnic.net`
			`- ARIN: http://ws.arin.net`
			`- LACNIC: http://www.lacnic.net`
			`- RIPE: http://www.ripe.net`
Update README.md 2020-08-11 19:28:55 +00:00
			`### BGP looking glasses`
Adding breached data search tools 2021-06-18 22:48:37 +00:00			`- BGP4: http://www.bgp4.as/looking-glasses`
			`- BPG6: http://lg.he.net/`
Update README.md 2020-08-11 19:28:55 +00:00
			`### DNS`
Update README.md 2023-06-30 14:12:58 +00:00			`- dnsenum - https://code.google.com/p/dnsenum`
			`- dnsmap: https://code.google.com/p/dnsmap`
			`- dnsrecon: https://www.darkoperator.com/tools-and-scripts`
			`- dnstracer: https://www.mavetju.org/unix/dnstracer.php`
			`- dnswalk: https://sourceforge.net/projects/dnswalk`
Adding breached data search tools 2021-06-18 22:48:37 +00:00
			`## The OSINT Framework`
			`- [OSINT Framework](https://osintframework.com)`

Update README.md 2020-08-11 19:28:55 +00:00
Update README.md 2021-02-15 01:27:18 +00:00			`## Dark Web OSINT Tools`
			`### Dark Web Search Engine Tools`
			`- [Ahmia Search Engine](https://ahmia.fi) and [their GitHub repo](https://github.com/ahmia/ahmia-site)`
			`- [DarkSearch](https://darksearch.io) and their [GitHub repo](https://github.com/thehappydinoa/DarkSearch)`
			`- [Katana](https://github.com/adnane-X-tebbaa/Katana)`
			`- [OnionSearch](https://github.com/megadose/OnionSearch)`
Update README.md 2021-02-23 18:49:01 +00:00			`- [Search Engines for Academic Research](https://www.itseducation.asia/deep-web.htm)`
adding OSINT tools 2021-02-24 01:29:10 +00:00			`- [DarkDump](https://github.com/josh0xA/darkdump)`
Update README.md 2021-02-15 01:27:18 +00:00
			`### Tools to Obtain Information of .onion Links`
			`- [H-Indexer](http://jncyepk6zbnosf4p.onion/onions.html)`
			`- [Hunchly](https://www.hunch.ly/darkweb-osint)`
			`- [Tor66 Fresh Onions](http://tor66sewebgixwhcqfnp5inzp5x5uohhdy3kvtnyfxc2e5mxiuh34iid.onion/fresh)`

			`### Tools to scan onion links`
			`- [Onioff](https://github.com/k4m4/onioff)`
			`- [Onion-nmap](https://github.com/milesrichardson/docker-onion-nmap)`
			`- [Onionscan](https://github.com/s-rah/onionscan)`

			`### Tools to Crawl Dark Web Data`
			`- [TorBot](https://github.com/DedSecInside/TorBot)`
			`- [TorCrawl](https://github.com/MikeMeliz/TorCrawl.py)`
			`- [OnionIngestor](https://github.com/danieleperera/OnionIngestor)`

			`### Other Great Intelligence Gathering Sources and Tools`
Update README.md 2021-03-18 18:51:48 +00:00			`- Resources from Pentest-standard.org - http://www.pentest-standard.org/index.php/PTES_Technical_Guidelines#Intelligence_Gathering`
updating passive recon tools 2018-08-07 20:34:22 +00:00
Update README.md 2020-08-11 19:28:55 +00:00			`### Active Recon`
Update README.md 2021-03-18 18:51:48 +00:00			`- Tons of references to scanners and vulnerability management software for active reconnaissance - http://www.pentest-standard.org/index.php/PTES_Technical_Guidelines#Vulnerability_Analysis`