Sindbad~EG File Manager
3
�uGh�o � " @ s d dl mZ d dlZd dlZejd,krzyd dlmZ W n ek
rN eZY nX d dl Z d dl
Z d dl m
Z
mZ d dlZn(d dlZd dlZd dlZd dl
m
Z
mZ ejd-kr�d dlZnd dlZd dlZd dlZd dlZy d dlZW n ek
�r d dlZY nX d dlZd dl Z d dl Z d dl Z d dl Z d dl Z d dl!Z!d dl"Z"d dl#Z#d dl$m%Z% y0d dl&m'Z'm(Z( d d l)m*Z*m+Z+ d d
l,m-Z- W nB ek
�r� d d l'm'Z' d d l.m(Z( d d
l/m*Z*m+Z+m-Z- Y nX dZ0e j1j2e j1j3e4��Z5e j1j6dd�e j1j6dd�e j1j6dd�e j1j6e j1j2e5�d�e j1j6e j1j2e5�d�e j1j6e j1j2e5�d�e j1j3d�e j1j3d�e j1j3d�f Z7e j8dk�r�e j1j6e5ddd�e j1j6e5ddd�e j1j6e5ddd�fZ7dd� Z9d.dd�Z:dd � Z;d d � Z<d d � Z=d!d"� Z>d#d$� Z?g fd%d&�Z@G d'd(� d(�ZAd)d*� ZBeCd+k�reB� dS )/� )�print_functionN� )�
basestring)�Queue�Empty� )� OptionParser)�urlparse� urlencode)�urlopen�Request)� HTTPError)r )r
)r r r
z1.3.1z/etcz agent360.inizagent360-custom.inizagent360-token.ini�ntz..�configc C sP t dd�} | j� }| jdd�}djdt ddj|� d | d
| jjd d � f�S )
z�
Return string with info about agent360:
- version
- plugins enabled
- absolute path to plugin directory
- server id from configuration file
T)� dry_instance�enabled)�state�
z Version: %szPlugins enabled: %sz, zPlugins directory: %sz
Server: %s�agent�server)�Agent�_get_plugins_path� _get_plugins�join� __version__r �get)r � plugins_pathZplugins_enabled� r �:/root/tmp/pip-build-gzoz1_uw/agent360/agent360/agent360.py�infoM s
r �httpsc C s� t � }|jdddd� |jddtddd � |j� \}}|d }td
d �}t|�d kr^|d }ntjjt d
�}t|�dkr�|d }nd}|j
d kr�d}n|j
}|j d kr�djt � �} nd} d|kr�|j
d�d }
|j
d�d }npytj� d } W n tk
�r tj� } Y nX t| d |jjdd� d t|| ||| d��jd�d�j� j� }
t|
�dk�r�td|
� t|d�jd ||
f � n td |
� d S ) Nz-tz--tagsz Comma-separated list of tags)�helpz-az --automonr z6Enable/disable automatic monitoring of hosted websites)�type�defaultr! T)r � zagent360-token.ini� � �,�_z://�data�hello_api_hostz/hello)�user�hostname� unique_id�tags�domainszutf-8)r) � zGot server_id: %s�wz [DEFAULT]
user=%s
server=%s
z Could not retrieve server_id: %s) r �
add_option�int�
parse_argsr �len�os�pathr �__FILEABSDIRNAME__r. Zautomon� _get_domains�split�uname�AttributeError�socket�getfqdnr r r r
�encode�read�decode�print�open�write) �proto�parser�options�argsZuser_idr Ztoken_filenamer- r. r/ Z server_idr, r r r �hellob sN
rI c C sl g } yNt jddg�}x:|j� j� D ]*}d|kr0q"|j� jd�}| j|d � q"W W n tk
rf Y nX | S )NZ apachectlz-SZ namevhost� r )�
subprocess� check_outputrA �
splitlines�stripr: �append�FileNotFoundError)r/ �output�line�colsr r r �_get_apache_domains� s rT c C s� g } ynt jddg�}xZ|j� j� D ]J}d|kr0q"|j� jd�}t|�dkr"|d jdd�jd d�}| j|� q"W W n t k
r� Y nX | S )
NZnginxz-TZ server_namerJ r% r$ �;r& �")
rK rL rA rM rN r: r5 �replacerO rP )r/ rQ rR rS �domainr r r �_get_nginx_domains� s rY c C s g } yBt jdddg�}tj|�}x"|d d D ]}| j|d � q.W W n� tk
r� y6t jdddd g�}x |j� j� D ]}| j|� qxW W nj tk
r� xTtt t
� t � ��D ]<}d
|kr�q�|j d �r�q�|j d �r�q�|j d
�r�q�| j|� q�W Y nX Y nX | S )NZwhmapi1z--output=jsonprettyZget_domain_infor) r/ rX Zplesk�binz--list�.z .localdomainz
.localhostz.local)
rK rL �json�loadsrO rP rA rM �list�setrT rY �endswith)r/ Zjson_str�responserX �strr r r r9 � s.
r9 c C s t tt� �� d S )N)rB r5 r9 r r r r �
count_domains� s rc c C s0 t | t�r&tjj| �}tjj|�d S | jS d S )Nr )�
isinstancer r6 r7 �basename�splitext�__name__)�pluginre r r r � _plugin_name� s
ri c 3 C s� t dd�}|j� }|tjkr*tjjd|� | sL|jdd�} tddj| � � �xZ| D �]P}td| � y,tjdkr�t j
j |�}nt j
|�\}}}W n. tk
r� } ztd |� wTW Y d d }~X nX zny8tjdkr�t j
j|�} |jj| � nt j||||�} W n0 tk
�r, } ztd
|� wTW Y d d }~X nX W d tjdk �rL|�rL|j� X y(| j� j|j�}
ttj|
d
dd�� W qT tk
�r� } ztd|� W Y d d }~X qTX qTW d S )z�
Test specified plugins and print their data output after single check.
If plugins list is empty test all enabled plugins.
T)r r r )r z Check all enabled plugins: %sz, z%s:r r z Find error:Nz Load error:)�indent� sort_keyszExecution error:)r r )r r )r r )r r �sysr7 �insertr rB r � version_info� importlib�util� find_spec�imp� find_module� Exception�module_from_spec�loader� exec_module� load_module�close�Plugin�runr r\ �dumps) �pluginsr r � plugin_name�spec�fp�pathname� description�e�module�payloadr r r � test_plugins� s@
r� c @ s� e Zd Ze� Ze� Ze� Ze� ZdZd dd�Z dd� Z
dd� Z dd � Z d
d � Z
d d
� Zdd� Zdd� Zdd� Zdd� Zdd� Zd!dd�Zdd � Zd d � Zd S )"r Fc C s: | j � i | _|rdS | j� | j� | j� | j� dS )z0
Initialize internal strictures
N)� _config_init�
plugins_cache�
_logging_init�
_plugins_init�_data_worker_init� _dump_config)�selfr r r r �__init__" s zAgent.__init__c C s2 t jdkr t jj| jjdd��S | jjdd�S d S )Nr r r} )r6 �namer7 �
expandvarsr r )r� r r r r 3 s
zAgent._get_plugins_pathc C s� ddt jdddtjjtd�ddddddd d
d dd �}d
ddg}tjdkrTtj |�}n
t
j |�}|j t � || _
xF|D ]>}| j|� |dkr�| j
j|dd� |d
krt| j
j|dd� qtW dS )z1
Initialize configuration object
�< �
�d r} �nor& zingest.monitoring360.iozapi.monitoring360.ioz/v2/server/pollz/var/log/agent360.log�a)�
max_data_span� max_data_age�
logging_level�threads�ttl�intervalr} r rK r+ r �api_hostr* �api_path�log_file�
log_file_mode�max_cached_collectionsr � executionr) r r� r$ g �?NiX )r )�logging�INFOr6 r7 r r8 rl rn � configparser�RawConfigParser� ConfigParserr@ � ini_filesr �_config_section_creater_ )r� �defaults�sectionsr �sectionr r r r� : s>
zAgent._config_initc C s | j j|�s| j j|� dS )zc
Create an addition section in the configuration object
if it's not exists
N)r � has_section� add_section)r� r� r r r r� c s z Agent._config_section_createc C s� | j jdd�}tjdkr0tjj| j jdd��}n| j jdd�}| j jdd�}|dkrVn |dkrdd}n|d krrd}nd}|d
kr�tj|d � n\ytj|||d d
� W nD t k
r� } z(tj|d � tj
d|� tj
d� W Y dd}~X nX tj
d|� dS )z,
Initialize logging faculty
r r� r r� r� r1 r� �truncaterO �-)�levelz,%(asctime)-15s %(levelname)s %(message)s)�filename�filemoder� �formatz IOError: %szDrop logging to stderrNzAgent logging_level %i)r1 r� ) r �getintr6 r� r7 r� r r� � basicConfig�IOErrorr )r� r� r� r� r� r r r r� k s*
zAgent._logging_initc
C sr t jd� | j� }tjtjj|d��}|tjkr>tjjd|� i | _ �x&|D �] }t
|�}|dkrdqL| j |� | j j
|d�rL| j j
|d�r�d| j |< qLtjd kr�tjj|�}ntj|�\}}}zpy8tjd
kr�tjj|�} |jj| � ntj||||�} W n2 tk
�r* d } t jd
ttj� d �� Y nX W d tjdk �rJ|�rJ|j � X | �r^d| j | < qLt jd |� qLW d S )z&
Discover the plugins
r� z*.pyr r} r rK r r Nzimport_plugin_exception:%szimport_plugin:%s)r r )r r )r r ) r� r r �globr6 r7 r rl rm �scheduleri r� r �
getbooleanrn ro rp rq rr rs ru rv rw rx rt �errorrb �exc_infory )
r� r � filenamesr� r� r r� r� r� r� r r r r� � s>
"
zAgent._plugins_initc
C s( t jtj|ft jt jdd�}tjdtj� |j � | j
j dd�}t |�}| j
j |d�}|| p^d}|j
� xB|jdkr�|d kr�tjd
tj� |� tj|� |d8 }|j
� qjW |jdkr�tjd tj� |j � tj|j tj� |j� \}}|jd ks�|�rtjd tj� ||j|� |�r tj|�} nd} | S )
z0
Execute /task/ in a subprocess
T)�stdout�stderr�universal_newlinesz
%s:process:%ir� r� r� r$ Nr z
%s:tick:%iz
%s:kill:%iz %s:%s:%s:%s)rK �Popenrl �
executable�PIPEr� �debug� threading�
currentThread�pidr r� ri �poll�
returncode�time�sleepr� r6 �kill�signal�SIGTERM� communicate�pickler] )
r� �task�processr� r� r� Zticksr� r� �retr r r �_subprocess_execution� s2
zAgent._subprocess_executionc C sb �x@| j r tjdtj� � P tjdtj� | jj� � y| jj� }W n t k
rX P Y nX tjdtj� |� t
|�}y| j j |d�}W n d}Y nX t
j
� }t|t�r�| j|�}nry<| jj|| jj |g �i� |j| j| d�}|j| j �}W n4 tk
�r& tjd� dttj� d �i}Y nX | jj|||||d
�� qW | j jtj� � | j j � d S ) zW
Take queued execution requests, execute plugins and queue the results
z %s:shutdownz%s:exec_queue:%iz
%s:task:%sr� r� )Z agent_cacheZplugin_exception� exceptionr )�tsr� r� r� r� N) �shutdownr� r r� r� r� �execute�qsize�
get_nowaitr ri r r r� rd r r� r� �updaterz r{ rt r� rb rl r� �metrics�put�cemetery�hire�release)r� r� r� r� r� r� rh r r r �
_execution� sD
zAgent._executionc C s t jdtj� � | jjdd�}| jjdd�}| jjdd�}| jjdd�}| jjdd�}| jjdd �}| jjdd
�}| jjdd �}g } g }
d } �x�| r�d
}n| jjdd�}tj� } | jr�t jdtj� � P t j dtj� | j
j � t |
�� xX| j
j � �r>y|
j
| j
j� � W q� tk
�r: }
zt jd|
� W Y dd}
~
X q�X q�W |
�r�tdd� |
D ��}tdd� |
D ��}tj� }d}|| |k�r�t j d� d }d}n || |k�r�t jd� d }d }|�r�d} dd||f d�}t j dtj|
d d d �� |�o�|�s
t jd � d }�n�zԐy@tjtj� d �}tjd0k�rBt j j ||d!d"�}nt j ||d!d"�}| �r�t jd#t | �� xj| �r�|j d$d%|t!f | d& |d'� |j"� }|j#� |j$d(k�r�| d&= t j d)|j$� nt%d*|j$ ���qjW t jd+� |j d$d%|t!f t&j't(tj|
�d, �j)� �|d'� |j"� }|j#� |j$d(k�r@t j d)|j$� d }nt%d*|j$ ��W n� tk
�r� }
znt jd-|
� |d&k�r�t | �|k�r�| d&= t jd.|� t jd/� | j
t&j't(tj|
�d, �j)� �� g }
W Y dd}
~
X nX W d|j*� X |�r�g }
|tj� | }|d&kr�tj+|� q�W dS )1zA
Take and collect data, send and clean if needed
z%sr) r� r� r r� r� r r+ r� r� Tr� z %s:shutdownz %s:data_queue:%i:collection:%izData queue error: %sNc s s | ]}|d V qdS )r� Nr )�.0r� r r r � <genexpr>' s z Agent._data.<locals>.<genexpr>c s s | ]}|d V qdS )r� Nr )r� r� r r r r� ( s Fz
Max data spanz Max data agezapplication/jsonz ApiKey %s:%s)z Content-type�
Authorizationzcollection: %sr% )rj rk z&Empty server or user, nowhere to send.)�cafiler � )�context�timeoutz Sending cached collections: %i�PUTz
%s?version=%sr )�headers�� zSuccessful response: %szUnsuccessful response: %szAll cached collections sentr z Failed to submit collection: %szCReach max_cached_collections (%s): oldest cached collection droppedz,Cache current collection to resend next time)r ),r� r r� r� r r r� r� r� r� r) r� r5 rO r� rt r� �min�max�warningr\ r| �ssl�create_default_context�certifi�whererl rn �http�client�HTTPSConnection�httplib�requestr � getresponser@ �status�
ValueError�bz2�compressrb r? ry r� )r� r� r� Zmax_ageZmax_spanr r+ r� r� Zcached_collectionsZ
collectionZ initial_dataZloop_tsr� Zfirst_tsZlast_ts�now�send�cleanr� �ctx�
connectionra �sleep_intervalr r r �_data s� $
"
z Agent._datac C s t jd� tj| jd�j� dS )z/
Initialize data worker thread
r� )�targetN)r� r r� �Threadr �start)r� r r r r� x s
zAgent._data_worker_initc C s< t jdkrtj� }ntj� }| jj|� tjd|j� � dS )z,
Dumps configuration object
r z
Config: %sN)r ) rl rn �io�StringIOr rD r� r �getvalue)r� �bufr r r r� s
zAgent._dump_configr c C s� | j � }g }xztjtjj|d��D ]b}t|�}|dkr8q"| j|� |dkrd| jj|d�r�|j |� q"|dkr"| jj|d�s"|j |� q"W |S )z0
Return list with plugins names
z*.pyr} r �disabled)
r r� r6 r7 r ri r� r r� rO )r� r r r} r� r~ r r r r � s
zAgent._get_pluginsc C sV t jd| jj� � x>y| jj� }W n tk
r8 P Y nX t jd|� |j� qW dS )zb
Join with dead workers
Workaround for https://bugs.python.org/issue37788
z cemetery:%iz
joining:%sN)r� r� r� r� r� r r )r� �threadr r r �_rip� s z
Agent._ripc s� t jd� | jjdd�}tj| jjdd��| _�y��x�| j� t j � � t j
dtj � � xx| j j
� r�| j j� }|d }t j
d|� |jd �}|r�t� �| jj|d� | j|< t|tj�r�|j|d < | jj|� qXW � fd
d �| jj� D �}x�|D ]�}t j
d |� | j|= | jj|� | jjd
��rzy&tj| j d�}|j � t j
d|� W n0 t k
�rv } zt j d|� W Y dd}~X nX q�t j d� | j j� dd| jjdd�id�� q�W dt j � � }|dk�r�t j |� q6t j d� t j d� q6W W n� t!k
�r� t j t"j#� d � t jd� | j� d} x�| �r�tj$� }
t jd |
� d d � |
D �} | �sht jd � t"j%d� d| _&t jd t'| �� x&| D ] }t jd ||� |j(|� �q�W �q&W Y n2 t k
�r� } zt j)d!| � W Y dd}~X nX dS )"z.
Start all the worker threads
zAgent main loopr r� r� r� z
%i threadsr� z
metrics:%sr� c s g | ]\}}|� kr|�qS r r )r� �what�when)r� r r �
<listcomp>� s z Agent.run.<locals>.<listcomp>z
scheduling:%sF)r z new_execution_worker_thread:%sz Can not start new thread: %sNZthreads_cappedZagent_internalZthreads_capping)r� r� r� g �?r z'not enough time to start worker threadsg�������?z
Shutting downTzRemaining threads: %sc S s( g | ] }|j � rt|tj� r|�qS r )�isDaemonrd r� � _MainThread)r� r r r r r � s
zBye!z Waiting for %i threads to exitzJoining with %s/%fzWorker error: %s)*r� r r �getfloatr� � Semaphorer� r� r r� r� � activeCountr� r� r� r r3 r� rd �types�
ModuleType�__file__r) r� �itemsr� �acquirer r� r rt r� r� �KeyboardInterruptrl r� � enumerate�exitr� r5 r r� ) r� r� r� r� rh r� r r� r �wait_forZ all_threadsr )r� r r{ � s|
z Agent.runN)F)r )rg �
__module__� __qualname__r r� r� r) r� r� r� r r� r� r� r� r� r� r r� r� r r r{ r r r r r s&
)!, /r
r c C sl t tj�dk�r^tjd jd�r8tjd dd � tjd< tjd dkr^tdjd�� tj� n�tjd d kr�tt� � tj� n�tjd d
kr�tt� tj� n�tjd d kr�tjd= tjt � � n�tjd d kr�tjd= tjt
� � nttjd d
k�rtjd= tjt dd�� nJtjd dk�r<tjt tjdd � �� n tdtjd tj d� tjd� n
t
� j� d S )Nr$ z--r% r! r �!Run without options to run agent.�,Acceptable options (leading -- is optional):�4 help, info, version, hello, insecure-hello, testr �versionrI z
count-domainszinsecure-hellor� )rE �testzInvalid option:)�file)r r! r" )r5 rl �argv�
startswithrB r r r r rI rc r� r� r r{ r r r r �main s:
r( �__main__)r )r r )r )D�
__future__r r� rl rn Z
past.builtinsr � ImportErrorrb r� � http.clientr� �queuer r r r� r� r �importlib.utilro rr r� r� r� r\ Z
simplejsonr� r6 r� r� r= rK r� r� r �optparser � urllib.parser r
�urllib.requestr r � urllib.errorr
�urllib�urllib2r r7 �dirname�abspathr r8 r r� r� r rI rT rY r9 rc ri r� r r( rg r r r r �<module> s�
8! 0 i$
Sindbad File Manager Version 1.0, Coded By Sindbad EG ~ The Terrorists