Account suspended 2

[Pedja]

We allowed the user access to retrieve the data AND gave him access to limit his forums accordingly once a ticket was finally submitted.

This is good and I think it solves all "after shutdown" concerns.

Now we have to find a way to monitor situation to avoid to be suspended at all.
Unfortunately, my experience with system administration is limited but, it's never late to learn more

[DomainDog]

Sorry to paraphrase myself, but in a humorous way, this sums up the problem:

The inability to know when one is approaching violation is why this question stays alive.

Another scenario:

ServerAdmin: "Don't overtax system resource limits!"
ConfusedYou: "What's the system resource limits?"
ServerAdmin: "When you're using too much!"
ConfusedYou: "How will I know when I'm using too much?"
ServerAdmin: "When your account is suspended!"

Not very helpful!

I think most everyone is perfectly willing to respect the limits. The problem seems to be, no one can tell us exactly where those limits are, and how to watch them.

I doubt any of this was caused by maliciousness, only ignorance.

Woof!

[Detonator]

To find out HOW MANY processes your site is currently running (each concurrent user clicking something on your site creates 1 process), you use the command

watch 'pstree -u | grep httpd'

(include the singlequotes). It reads what user currently has what number of webserver/php processes running. Output will be refreshed every 2 seconds and looks like this:

httpd78*[httpd(nobody)]
            2*[httpd(nobody)httpd]
            httpd(nobody)php(hijjaz)
            httpd(nobody)php(ringtone)
            httpd(nobody)php(logi)
            httpd(nobody)php(myindian)
            3*httpd(nobody)php(schwarzt)
            httpd(nobody)php(bananko)

This means there are 10 people clicking links at the moment.. Of these 10 people, 8 people are clicking links on php websites.. The owners of these php-websites are shown as you can see.

When someone's website overloads the server, you can easily see who it is, and whether or not it's you.. You can easily tell WHEN the server is overloaded - it's very slow. In that case login via SSH and check who's the cause.

[Jonathan]

hmm but how many processes (PHP)
before we go and get shutdown? lol

[AndrewT]

hmm but how many processes (PHP)
before we go and get shutdown? lol

If you consistently have 10 or more processes going, that's likely going to end up being an issue (possibly less depending on the situation/content/etc.). PHP scripts are processed very fast so if you always have 10 of them running at the same time either it is a very poor script or you have quite a few visitors loading the scripts.

[Pedja]

I played today with ps and pstree and I am not happy. Ok, I can see what accunts are used for php scripts, but what about sites without PHP, they are all shown as nobody user no matter what accoutn is used.

On the other hand, it is lesslikely that static html site will produce some problems.

Also, I used SHOW PROCESSLIST comand in mysqld to see number of active MySQL processes, which also may lead to number of user sessions active on site.

I do not know how to see CPU and memory resources used by site account.

[reviewum.com]

To find out HOW MANY processes your site is currently running (each concurrent user clicking something on your site creates 1 process), you use the command

watch 'pstree -u | grep httpd'

OKAY!! Now we're talkin'!!!

So, if one were to run this exact command in ssh then it will tell them all of the processes running and by whom? I don't have to put in a specific user or anything?

A few Questions:

1) How the heck do I exit out of this after I've seen what I need?
2) I've read in the past that on a shared server many of the scripts, etc. are run by root or "nobody" so it will be hard to tell who's hogging all the resources. Is this true with this command and with the TOP command?

Here's my test: Server status on my server says we're at "16.38 (1 cpu)". I ran the command: watch 'pstree -u | grep httpd' but there are only a few lines every now and then and they all say nobody.

| |-httpd(nobody)---php(32532)
| |-httpd(nobody)---php(32552)
| `-httpd(nobody)---php(32431)

I then did a TOP command and got:

492 mailnull 18 0 2444 2360 1808 R 3.9 0.1 0:00 exim
15 root 9 0 0 0 0 SW 1.9 0.0 110:39 kjournald
547 #32532 18 0 5028 5028 3300 R 1.9 0.3 0:00 php
545 #32383 12 0 3128 2564 1136 S 1.3 0.1 0:00 cppop


So, did I learn anything helpful about who and where the resource usage is coming from to cause my server to say it is at "16.38 (1 cpu)"???
Thanks again! Keep the resource monitoring suggestions coming!

[DomainDog]

O
| |-httpd(nobody)---php(32532)
| |-httpd(nobody)---php(32552)
| `-httpd(nobody)---php(32431)

I then did a TOP command and got:

492 mailnull 18 0 2444 2360 1808 R 3.9 0.1 0:00 exim
15 root 9 0 0 0 0 SW 1.9 0.0 110:39 kjournald
547 #32532 18 0 5028 5028 3300 R 1.9 0.3 0:00 php
545 #32383 12 0 3128 2564 1136 S 1.3 0.1 0:00 cppop

So...... I pasted this into Altavista's BabelFish translator, and here's what it returned:

1010 1001 1011 1001 WTF???
1011 1010 1010 0011 No Sprekenze
1001 1011 1001 1011 Jabberwocky
FFC0 AEEC 0DDE FFFF !^&@&*^#^!

Maybe someone could put this into our newly reconstituted FAQ in intelligible format, for all the non-MIT graduates?

[reviewum.com]

I've got a question... can our trouble ticket support friends (other than Andrew) look up server resource usage for us?

When we notice one of our sites is at a peak usage time can we send in a ticket and ask: "Hey support friend, my site www.mysite.com on server CP89 is getting a lot of use right now... could you check out the server usage to see if I'm far below the radar, or in the danger zone?"

I think this is one of the most pro-active things we can do... only problem is, if someone has a site that they are worried about, they probably won't want to say, "Hey Andrew... here's my really busy site, could you please suspend it??"

So, this leaves us back at the tools we can use ourselves to check on resource usage.

Where are all those cool unix / linux geeks... er... um... professionals when you need them?

[Detonator]

You cannot really monitor the server when you're not an administrator. Also, you need to know what you are looking for when doing monitoring.. The essence in this forumthread that you only need to monitor your OWN services to see whether YOU yourself are the cause of overloading the server.

On unix, every user has a userid ('UID'). On some of Dathorn's servers, you can only watch UID's, and not usernames, running certain services.. In my previous posting, pstree showed usernames in it's output. In reviewum.com's post it only shows UIDs:

| |-httpd(nobody)---php(32532)
| |-httpd(nobody)---php(32552)
| `-httpd(nobody)---php(32431)

This snippet of output means the httpd-server is run as user "nobody" (this has security reasons, not our concerns here), while it is serving phpscripts owned by UIDs 32532, 32552 and 32431. So it's serving 3 websites at the moment, belonging to 3 different people.

Because we are only interested in our own websites, you only need to know what your own UID is. Just use this command:

ls -l --numeric ~

This shows a listing of your homedirectory. The third column shows your own UID.. So now you only need to watch for processes with this UID since they belong to you You always keep the same UID, you are simply a number on the system.

Another issue, as mentioned by someone else.. Some processes run as a generic user, not as a specific user. Important ones include:

.- process "mysqld", runs as user "mysqld"; this is mysql server
.- process "exim", runs as user "mailnull"; this is the outgoing mailserver (SMTP)

exim, the outgoing mailserver, often uses 99.9% cpuload when someone is sending out a lot of emails.. For example for a mailinglist. Well, you probably know when you are running a mailinglist, in that case take an eye on the serverload (the exim process in particular) when you are sending out your daily digests.

mysqld processes usually run in combination with a PHP or perl/cgi script.. When you use the "top" command and see one PHP script taking a lot of load, and you see a mysqld process taking a lot of load at the same time, then you should count their load together.. For example, if the PHP script uses 3% cpuload, and every time this script is executed a mysqld process shows up taking 20% cpuload, then the combination makes 23%. Some guesswork is involved here, but there's no such thing as coincidence.

Just apply some logic here. Watch the system for 5-10 minutes when it's slow, to get a good overview of what is going on.. There's no simple oneway solution.

To avoid a suspended account, just make sure it's not YOUR processes showing up in the ps/pstree/top listings during server-slowness.. If you are the one with the most/heaviest processes when the server is acting slow, then you're the most likely candidate for a suspension

[reviewum.com]

Detonator,

You're my freakin' NEW BEST FRIEND!!!

That is the kind of post I've been waiting for, for about 3 years!!!

So, to recap, I'm looking for

1) php processes run by my username or userID# that are taking lots-o-resources (so, is CPU and/or Memory usage more/less important or the same?)

2) mysqld processes running in conjunction with the php processes listed above.

3) exim running high when I (or a script) is sending out a bunch of email.

Anybody else got any information that is as good as what Detonator just posted???

Hey KyleC, talk about FAQ.... Is there a topic more discussed than this one???


oh... by the way, what does this mean:

   |-httpd---29*[httpd(nobody)]

29 nobody processes???

[Detonator]

pstree shows a tree of running processes on the server. Every process is merely a leaf in a big processtree. The root-process is called "init" on a linux machine, everything else is below that one. The Apache webserver also runs itself in a tree-shape, so below the first httpd process are other httpd processes.

init-+
     |-httpd---20*[httpd(nobody)]

This is what the situation looks like when Apache is not serving any pages. You see the init-process on top (the mother of all linux processes), and httpd below that one, without a specific username. When a username is not shown in pstree, it means the command is run by the root-user. So the above is actually the same as:

init(root)-+
     |-httpd(root)---20*[httpd(nobody)]

The init-process launched the webserver process (httpd), running as root-user. The webserver itself then started 20 subprocesses (httpd) waiting for 20 requests, running as user nobody for securityreasons. When a hacker hacks a webserver, through a securityhole usually called "buffer overflow" or "heap overflow", the hacker will become the user which was running the webserver.. That's why the Apache webserver launches itself in a tree-shape; when a hacker breaks security, the hacker will have access to the entire server as user "nobody", which is useless.. When the hacker would've gained access as user "root", he could've destroyed the entire server.

In the example above, Apache opened 20 processes listening for an incoming connection.. So even without anyone requesting webpages, apache has 20 processes idle.. Because Apache is smart, it automatically increases the number of spare-processes when the load of the server increases, and it decreases when the load of the server goes down.. So sometimes it shows 20, sometimes it shows 29 (as in your question), and sometimes it shows much much more. These are simply processes which are doing nothing, but which are always ready for action.

When a spare-process gets something to do, a new branch in the tree will show up with a username.. Not all webservers show the exact same characteristic, only servers running phpsuexec. Servers without phpsuexec list every webrequest as owned by "nobody", which is pretty useless on a shared webserver.

So, in short, for the listing I posted earlier:

httpd78*[httpd(nobody)]
            2*[httpd(nobody)httpd]
            httpd(nobody)php(hijjaz)
            httpd(nobody)php(ringtone)
            httpd(nobody)php(logi)
            httpd(nobody)php(myindian)
            3*httpd(nobody)php(schwarzt)
            httpd(nobody)php(bananko)

This means there is one big parent httpd-process, with 78 children listening for connections. Of these 78 children, 2+1+1+1+1+3+1=10 are currently serving a webpage.. 2 of these are serving plain old html files, 8 are serving php scripts.

[reviewum.com]

Once agan Detonator I bow before thee!!!

Great posts! Thank you so very much...

You need to create a website out of all this good info.... i know there are probably about a million potential visitors from WHT and other places!

Thanks again!

[DomainDog]

You need to create a website out of all this good info.... i know there are probably about a million potential visitors from WHT and other places!

Um...... reviewum? I imagine that would get his account suspended! (just kidding!)

[reviewum.com]

Doh.... good point! We need to re-align our goals!!!

Hey support, how many hits can Detonator get to his cool new websites before they get suspended?

The sites are as follows:

https://www.I-dont-wanna-get-suspended-no-more.com

which would be parked on top of:

https://www.how-to-know-the-bomb-is-gonna-drop-on-my-site-before-it-really-happens.com

Thanks in advance!