Discussions  >  CollabNet Subversion Admins  >  httpd.exe Process Consumes all Spare CPU Cycles

Hide all messages in topic

All messages in topic

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author valentijnscholten
Full name Valentijn Scholten
Date 2012-03-03 09:39:02 PST
Message >
> FWIW, an issue like this should probably be viewable by looking at the "access" log or "subversion" log and seeing an unusually high amount of activity from one client.
>
For me the cpu problem is not related to requests coming in, so I have to find another solution.

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2012-02-27 09:48:27 PST
Message On Feb 27, 2012, at 12:45 PM, Paul Fruia wrote:

> Yes, we did resolve the issue. It turned out that the "Versions" SVN client for Mac OS X was constantly requesting data from the SVN server. I don't use Versions personally, but I some of our developers do, so I had them change the settings to disable automatic repo syncing. It's simple enough to go into Versions and do a comparison between the local repo and the SVN server without the need to keep it up to date automatically.
>
> Anyway, please check your SVN clients and make sure that they are not automatically syncing the local repos with the remote repos. This should fix the issue for you.

FWIW, an issue like this should probably be viewable by looking at the "access" log or "subversion" log and seeing an unusually high amount of activity from one client.

Mark

Re: Re: httpd.exe Process Consumes all Spare CPU Cycles Re: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author pfruia
Full name Paul Fruia
Date 2012-02-27 09:45:08 PST
Message Valentijn,

Yes, we did resolve the issue. It turned out that the "Versions" SVN
client for Mac OS X was constantly requesting data from the SVN server. I
don't use Versions personally, but I some of our developers do, so I had
them change the settings to disable automatic repo syncing. It's simple
enough to go into Versions and do a comparison between the local repo and
the SVN server without the need to keep it up to date automatically.

Anyway, please check your SVN clients and make sure that they are not
automatically syncing the local repos with the remote repos. This should
fix the issue for you.

I hope this help!

Thanks,
-- Paul

On Sun, Feb 26, 2012 at 12:46 PM, Valentijn Scholten <
valentijnscholten@ho​tmail.com> wrote:

> >
> > I'm still having trouble with this issue and it is impacting
> productivity. I have looked at the logs and the build scripts and nothing
> is jumping out at me.
> >
> Did you manage to resolve the issue?
> I notice this thread ends in mentioning it was solved, but that was by
> another user, or is that you?
>
> I am having the same problem.
>
> It started occuring somehwere after upgrading to subversion edge 2.0. I am
> not saying it is related to svn egde, but maybe to the subversion versions
> bundled with svn edge.
>



--
-- Paul

===================
Paul Fruia
Softeq Development Corp.
www.Softeq.com
(281) 552-5020
Attachments
attachment-517041

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author valentijnscholten
Full name Valentijn Scholten
Date 2012-02-26 10:46:06 PST
Message >
> I'm still having trouble with this issue and it is impacting productivity. I have looked at the logs and the build scripts and nothing is jumping out at me.
>
Did you manage to resolve the issue?
I notice this thread ends in mentioning it was solved, but that was by another user, or is that you?

I am having the same problem.

It started occuring somehwere after upgrading to subversion edge 2.0. I am not saying it is related to svn egde, but maybe to the subversion versions bundled with svn edge.

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-10-06 19:10:16 PDT
Message There were zero changes to Subversion in the update. Just a small change in a JavaScript that runs in your browser when installing the updates for the next release.

Sent from my iPhone

On Oct 6, 2011, at 9:51 PM, "mark hosang" <hosang at honest dot co dot jp> wrote:

> I started having this issue today after I updated our subversion server to the latest release 2.0.1. Our server gets the httpd apache process pinned at 100%. We have a vanilla install of collabnet subversion with no modifications to the apache configuration.
>
> To reiterate, subversion was perfectly fine on the box until we upgraded to 2.0.1 from 2.0.0 via the automatic upgrade tool in the admin page. Which would lead me to believe this is a bug in 2.0.1
>
> --------------------​--------------------​--------------
> http://subversion.op​en.collab.net/ds/vie​wMessage.do?dsForumI​d=3&dsMessageId=​430967
>
> To cancel your subscription to this CollabNet Subversion Admins discussion, please e-mail forum3-unsubscribe@s​ubversion.open.colla​b.net.

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author squarebox
Full name mark hosang
Date 2011-10-06 18:58:47 PDT
Message seems that the box wasn't restarted after upgrading to 2.0.1. We finally got permission to reboot the server and this has seemed to fix the issue for now.

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author squarebox
Full name mark hosang
Date 2011-10-06 18:51:48 PDT
Message I started having this issue today after I updated our subversion server to the latest release 2.0.1. Our server gets the httpd apache process pinned at 100%. We have a vanilla install of collabnet subversion with no modifications to the apache configuration.

To reiterate, subversion was perfectly fine on the box until we upgraded to 2.0.1 from 2.0.0 via the automatic upgrade tool in the admin page. Which would lead me to believe this is a bug in 2.0.1

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-10-05 10:53:49 PDT
Message On 10/5/11 1:28 PM, "Paul Fruia" <Paul dot Fruia at Softeq dot com> wrote:

> Actually, maybe a first step is to make sure that the hardware I'm running on
> is adequate. Can you please provide or direct me to a link showing the
> hardware specifications?

Subversion does not require a lot of hardware so we do not generally publish
specifications. We host our larger customers on pretty beefy Intel servers
but that is for over 10K users and several million SVN transactions per day
(plus other stuff).

Subversion itself is not multi-threaded but the Apache server can service
more requests with more threads so a multi-core server is a good idea. A
couple GB of RAM is good as is having your repositories on fast disk. In
general, I would not recommend putting the repositories on network.

We also generally find that Apache runs better on Linux than on Windows but
I usually recommend users choose the server OS based on what they are most
experienced managing. We have had a couple customers that ultimately moved
their server from Windows to Linux to get better results. To be honest, our
support team would have better information on this than I do.

Mark

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-10-05 10:48:31 PDT
Message On 10/5/11 1:25 PM, "Paul Fruia" <Paul dot Fruia at Softeq dot com> wrote:

> I'm still having trouble with this issue and it is impacting productivity. I
> have looked at the logs and the build scripts and nothing is jumping out at
> me.
>
> Can you please tell me what support options are available with CollaNet to
> help get this issue resolved?

Information on CollabNet Support is available here:

http://www.open.coll​ab.net/support/

Mark

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author pfruia
Full name Paul Fruia
Date 2011-10-05 10:28:33 PDT
Message Mark,

Actually, maybe a first step is to make sure that the hardware I'm running on is adequate. Can you please provide or direct me to a link showing the hardware specifications?

Thanks,
-- Paul

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author pfruia
Full name Paul Fruia
Date 2011-10-05 10:25:06 PDT
Message Hey Mark,

I'm still having trouble with this issue and it is impacting productivity. I have looked at the logs and the build scripts and nothing is jumping out at me.

Can you please tell me what support options are available with CollaNet to help get this issue resolved?

Thanks,
-- Paul

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-09-29 10:32:46 PDT
Message They each include:

"This applies to processes forked off from Apache children servicing
requests, not the Apache children themselves."

It also appears that Rlimit is a Linux or Unix-specific thing:

http://linux.about.c​om/library/cmd/blcmd​l2_setrlimit.htm



On 9/29/11 1:28 PM, "Paul Fruia" <Paul dot Fruia at Softeq dot com> wrote:

> Mark,
>
> That's odd. The docs I saw on those directives didn't say anything about
> "forked children" or whether or not it worked on Windows. My server is
> Windows 2003 SP2.
>
> Anyway, they obviously don't make a difference because I still have the issue.
>
> Thanks,
> -- Paul
>
> --------------------​--------------------​--------------
> http://subversion.op​en.collab.net/ds/vie​wMessage.do?dsForumI​d=3&dsMessageId=​43
> 0064
>
> To cancel your subscription to this CollabNet Subversion Admins discussion,
> please e-mail forum3-unsubscribe@s​ubversion.open.colla​b.net.

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author pfruia
Full name Paul Fruia
Date 2011-09-29 10:28:59 PDT
Message Mark,

That's odd. The docs I saw on those directives didn't say anything about "forked children" or whether or not it worked on Windows. My server is Windows 2003 SP2.

Anyway, they obviously don't make a difference because I still have the issue.

Thanks,
-- Paul

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-09-29 10:22:16 PDT
Message On 9/29/11 1:19 PM, "Mark Phippard" <mphippard at collab dot net> wrote:

> On 9/29/11 1:18 PM, "Paul Fruia" <Paul dot Fruia at Softeq dot com> wrote:
>
>> One other note, the httpd.conf is essentially the stock file with the
>> following additions:
>>
>> # pf - Directives to limit CPU and memory usage
>> RLimitCPU 20 30
>> RLimitMEM 20000000
>> RLimitNPROC 4 8
>
> FWIW, I have never heard of any of those directives and never seen anyone
> suggest using them, so I doubt they help with SVN.

Your server is Windows right? The Apache docs for those directives talk
about forked children so that would not apply to Windows as that is not how
Apache works on Windows.

Mark

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-09-29 10:19:44 PDT
Message On 9/29/11 1:18 PM, "Paul Fruia" <Paul dot Fruia at Softeq dot com> wrote:

> One other note, the httpd.conf is essentially the stock file with the
> following additions:
>
> # pf - Directives to limit CPU and memory usage
> RLimitCPU 20 30
> RLimitMEM 20000000
> RLimitNPROC 4 8

FWIW, I have never heard of any of those directives and never seen anyone
suggest using them, so I doubt they help with SVN.

Mark

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author pfruia
Full name Paul Fruia
Date 2011-09-29 10:18:01 PDT
Message Mark,

Hmm... okay. There are only about 10,000 entries in the subversion log. Again, I'll need to check on automated build scripts with our development teams.

One other note, the httpd.conf is essentially the stock file with the following additions:

# pf - Directives to limit CPU and memory usage
RLimitCPU 20 30
RLimitMEM 20000000
RLimitNPROC 4 8

Thanks,
-- Paul

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-09-29 10:03:32 PDT
Message On 9/29/11 1:01 PM, "Paul Fruia" <Paul dot Fruia at Softeq dot com> wrote:

> I'll need to check with all our development teams to make sure they are
> properly formatting their build scripts. However, in looking at the logs, I'm
> seeing something on the order of 200,000 access log entries per day. Is this
> enough to bog down the server?
>
> Also, for the most part, these look like accesses from individual users, but
> I'm sure some of them are automated.

No, 200K is not too many but of course not all requests are created equal.
You could look at the Subversion log which gives a slightly higher-level
look at the requests.

Mark

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author pfruia
Full name Paul Fruia
Date 2011-09-29 10:01:01 PDT
Message Mark,

I'll need to check with all our development teams to make sure they are properly formatting their build scripts. However, in looking at the logs, I'm seeing something on the order of 200,000 access log entries per day. Is this enough to bog down the server?

Also, for the most part, these look like accesses from individual users, but I'm sure some of them are automated.

Thanks,
-- Paul

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-09-29 08:10:37 PDT
Message On 9/29/11 11:06 AM, "Paul Fruia" <Paul dot Fruia at Softeq dot com> wrote:

> Thanks for the quick response. I will check the access logs, however, this
> happens immediately after restarting the service. It does seem to take a bit
> longer to occur after a system re-boot(perhaps a few minutes), but just
> stopping and re-starting the service doesn't seem to resolve the problem.
>
> I did see the viewvc topic, but we use this very rarely. We almost always use
> an SVN client like TortoiseSVN. Any other idea?

You are using the stock Apache httpd.conf configuration that we write from
SVN Edge? You mentioned trying to add a directive, have you done anything
else?

In terms of taking a few minutes, I would still check the logs. Maybe you
have rogue process like a Continuous Integration server that is constantly
polling the server.

An Apache server does not use any CPU once it has started unless something
is making requests to it. The access logs can help us track down whether:

* There is a single specific request triggering high CPU load
* There is something generating a lot of requests to the server

We have seen cases where build scripts do something like:

$ svn log -r0:HEAD url://server/repos

On a big repository, this request is enormously expensive. Just changing
the request to this:

$ svn log -rHEAD:0 url://server/repos

Can make a massive difference. That is just an example of course.

Mark

RE: Re: httpd.exe Process Consumes all Spare CPU Cycles RE: Re: httpd.exe Process Consumes all Spare CPU Cycles

Author pfruia
Full name Paul Fruia
Date 2011-09-29 08:06:30 PDT
Message Mark,

Thanks for the quick response. I will check the access logs, however, this happens immediately after restarting the service. It does seem to take a bit longer to occur after a system re-boot(perhaps a few minutes), but just stopping and re-starting the service doesn't seem to resolve the problem.

I did see the viewvc topic, but we use this very rarely. We almost always use an SVN client like TortoiseSVN. Any other idea?

Thanks,
-- Paul

Re: httpd.exe Process Consumes all Spare CPU Cycles Re: httpd.exe Process Consumes all Spare CPU Cycles

Author markphip
Full name Mark Phippard
Date 2011-09-29 08:01:21 PDT
Message On 9/29/11 10:56 AM, "Paul Fruia" <Paul dot Fruia at Softeq dot com> wrote:

> I am running CollabNet Subversion Edge on a Windows Server 2003 SP2 box and
> the Apache Server process (CollabNet Subversion Server) will consume any free
> CPU cycles up to 100%, effectively hobbling the server. I know it is this
> process because stopping the service removes the process from Task Manager and
> frees up CPU cycles.
>
> I have tried updating the httpd.conf file to limit CPU usage using the
> following command:
>
> RLimitCPU 20 30
>
> but it doesn't seem to make any difference.
>
> I have seen some mention in the forum of high memory usage, but oddly enough
> that does not seem to be a problem in my instance.
>
> At any rate, please let me know if anyone else has experienced this issue and,
> if so, if you were able to resolve it.

I would check the access logs to get an idea what users are doing when the
CPU gets high. I assume that simple starting Apache does not use up all
CPU? It has to be due to the requests being made.

As you can see in this thread there are certainly instances where ViewVC
browsing can create high server load until the request completes:

http://subversion.op​en.collab.net/ds/vie​wMessage.do?dsForumI​d=3&dsMessageId=​
429950

Mark

httpd.exe Process Consumes all Spare CPU Cycles httpd.exe Process Consumes all Spare CPU Cycles

Author pfruia
Full name Paul Fruia
Date 2011-09-29 07:56:46 PDT
Message All,

I am running CollabNet Subversion Edge on a Windows Server 2003 SP2 box and the Apache Server process (CollabNet Subversion Server) will consume any free CPU cycles up to 100%, effectively hobbling the server. I know it is this process because stopping the service removes the process from Task Manager and frees up CPU cycles.

I have tried updating the httpd.conf file to limit CPU usage using the following command:

  RLimitCPU 20 30

but it doesn't seem to make any difference.

I have seen some mention in the forum of high memory usage, but oddly enough that does not seem to be a problem in my instance.

At any rate, please let me know if anyone else has experienced this issue and, if so, if you were able to resolve it.

Many Thanks,
-- Paul
Messages per page: